Computer Science and Engineering Efficiently Monitoring Top-k Pairs over Sliding Windows Presented By: Zhitao Shen 1 Joint work with Muhammad Aamir Cheema 1, Xuemin Lin 21, Wenjie Zhang 1, Haixun Wang 3 1 The University of New South Wales, Australia 2 East China Normal University 3 Microsoft Research Asia

2 Introduction Top-k Pairs Query: Given a scoring function score() that computes the score of a pair of objects, return k pairs of objects with the smallest scores. Examples: k closest pairs queries k furthest pairs queries Top-k Pairs against sliding windows Given a data stream, return top-k pairs among the most recent N objects. Applications Wireless sensor network, stock market, traffic monitoring and transaction monitoring

3 Motivation No existing work for general pairs queries over sliding windows Support arbitrary scoring functions. Example: Fraud detection over transaction streams –Query the transaction pairs that have small time difference but the locations are far away. Select a.id, b.id from trans a, trans b where a.id <> b.id and a.account = b.account order by |a.time - b.time| - dist(a.loc, b.loc) limit k window [24 hours] Select a.id, b.id from trans a, trans b where a.id <> b.id and a.account = b.account order by |a.time - b.time| - dist(a.loc, b.loc) limit k window [24 hours] :15:20New York$ :18:10L.A.$1000

4 Problem Definitions (Preliminaries) Sliding Windows –A sliding window contains most recent N objects of the data stream. –The number of pairs is N(N – 1) / 2 Sliding window of size 5 newer older o1o1 o2o2 o3o3 o4o4 o5o5 o6o6 o7o o0o0 Lower bound runtime cost : O(N) for each new object Lower bound storage cost : O(N) Age of an object: The age of a pair depends on the older object.

5 Contributions Unified framework First to study top-k pairs queries over sliding windows. Support arbitrarily complex scoring functions Support efficient queries for any window size n ≤ N and any k ≤ K Lower boundExpected cost for our algorithms Storage requirementO(N)O(N) + O(K log(N/K)) for each scoring function Skyband maintenance cost for each object O(N)O(N (log (log N) + log K)) Answering top-k pairsO(k)O(log(log n) + log K + k)

6 Preliminaries p1p1 p2p2 p4p4 p7p7 Age Score Map all the pairs to an age–score space Top-2 pairs K-skyband [Papadias et al., TODS05] keeps the minimum set for the candidate results. p 2 dominates p 5 because p 2.score < p 5.score and p 2 expires no later than p 5. Task1 : how we efficiently maintain the K-skyband Task2 : how we use the K- skyband to efficiently obtain top-k pairs against any sliding window n ≤ N p 1 (o 0, o 1 )  (p 1.age, p 1.score)  (1, 3) o1o1 o2o2 o3o3 o4o4 o0o0 p3p3 p5p5 p6p6 p8p8 p9p9 p Naive: O(N |SKB|) for checking all N-1 pairs Expected size of skyband is O(K log(N/K)) Our: O(N log|SKB|)

7 p1p1 p2p2 p3p3 p4p4 2-skyband Age Score p5p5 Efficient Skyband Maintenance Can we find a boundary between the skyband points and non-skyband points? K-staircase How can we efficiently compute the K-staircase and K-skyband? s1s1 Update the K-staircase and K-skyband in O(|SKB| log K)), Check if a pair is dominated by K-skyband in O(log |SKB|) time for each new pair by doing binary search. p5p5 K-staircase s1s1 s2s2 s2s2 p1p1 p6p6 p7p7

8 Window size = NAny window size = n < N Efficient Query Answering p3p3 p1p1 p5p5 p7p7 p8p8 2-skyband Age Score p6p6 p4p4 p2p2 Can we do better for any sliding window size n < N? Use Priority Search Tree to index the skyband points Self-balancing tree Efficient 3-sides range query 6 p1p1 3 p5p5 1 p7p7 4 p6p6 2 p8p8 9 p2p2 8 p3p3 5 p4p4 Priority Search Tree

9 Efficient Query Answering p3p3 p1p1 p5p5 p7p7 p8p8 2-skyband Age Score p6p6 p4p4 p2p2 Our contribution: Retrieve top-k pairs in the 1-sided range. An algorithm similar to post-order traversal costs O(log|SKB| + k) Any window size = n < N 6 p1p1 3 p5p5 1 p7p7 4 p6p6 2 p8p8 9 p2p2 8 p3p3 5 p4p4 Priority Search Tree

10 What else in the paper? Efficient continuous queries on the skyband. Continuously monitoring the top-k results for any fixed k (k ≤ K) and n (n ≤ N). Amortized O(k/n (log |SKB| + k)) time per update. Optimization on monotonic scoring functions. Handling the k-closest pairs, k-furthest pairs queries. Applying Threshold Algorithm on sorted lists Improving the number of considered pairs for each new object from N to (d+1) N d/(d+1) K 1/(d+1)

11 Experimental Settings Real dataset. –Sensor data in the Intel research lab –2.3 million records. Synthetic data. –Uniform, correlated and anti-correlated distributions. –2 million objects –Closest and furthest pairs in Manhattan distance

12 Experiments (Overall Cost on real data) SCase: our algorithm using K-staircase to maintain the skyband. Naïve: maintains kN pairs and sort them on their scores. LB: shows lower bound cost Varying K Varying N (in thousands)

13 Experiments (Query Answering) Linear: scan the skyband points to find the top-k pairs. Snapshot: our snapshot query algorithm. Continuous: our continuous query algorithm. LB: an algorithm to obtain top-k results in O(k) time. Varying K Varying |Q| (in thousands)

14 Conclusion: First to study a broad class of top-k pairs queries over sliding windows. We present efficient algorithms and show that the performance of our algorithm is reasonably close to the lower bound cost. We provide extensive experiment results on both real and synthetic data sets to show the efficiency and scalability of the proposed algorithms.

15 Question and Answer Thank You! Any Questions?

16 Related Work Top-k Query Processing Fagin’s Algorithm (FA), threshold Algorithm (TA), no-random access (NRA) Top-k Pairs Queries Processing k-closest pairs queries k-furthest pairs queries Top-k pairs queries [Cheema et al., ICDE’11] Data Stream Processing Top-k query processing over data stream [Mouratidis et al., SIGMOD’06] k-nearest neighbour queries [Böhm et al., ICDE’07]

17 Experiments (Skyband Maintenance algorithm) Basic: maintening algorithm without K-staircase SCase: our algorithm using K-staircase to maintain the skyband. TA: Optimized algorithm for monotonic scoring functions. LB: show lower bound cost # of attributesVarying K