Stabbing the Sky: Efficient Skyline Computation over Sliding Windows COMP9314 Lecture Notes

COMP9314Xuemin Outline Introduction n-of-N Queries (n 1, n 2 )-of-N Queries Performance Evaluation Conclusions

COMP9314Xuemin Skyline Skyline Query : Input: a set of points in d- dimensional space. Output: points not dominated by another point. (x 1, x 2, …, x d ) dominates (y 1, y 2, …, y d ) iff x i <=y i (1<=i<=d) & ∃ k, x k <y k.

COMP9314Xuemin Applications Multi-criteria decision making… Stock Trading Example: What are the top deals?

COMP9314Xuemin Skyline Query Over Sliding Window Stock Trading Example Top deals of a stock in the last 5 mins? last 4 mins, … Top deals of a stock in the last 10K deals? … Queries: n-of-N model (  n <= N): the most recent n elements (n 1, n 2 )-of-N model One-time queries Continuous queries

COMP9314Xuemin Challenges Insertions & deletions (possibly high speed). On-line information –memory requirement –processing speed Existing techniques do not support n-of-N: [Borzsonyi et al (ICDE01), Tan et al (VLDB01), Kossman et al (VLDB 02), Papadias et al (SIGMOD03), Kapoor (SIAM J. comp00)] –support the computation of whole dataset –O (n log d-2 n) for d >= 4 & O (n log n ) otherwise

COMP9314Xuemin Results n-of-N: keep N’ (N’  N) elements where N’ = O (log d N) if data distribution on each dimension is independent. a novel encoding scheme, with O (N’) space, leads to n- of-N query time O ( log N’ + s ) instead of O (n log d-2 n). a new trigger based technique for continuously processing an n-of-N query. –trigger update time: O ( log s). –result update time: O (logδ) where δ is a result change. (n 1, n 2 )-of-N: similar results.

COMP9314Xuemin n-of-N Queries e is redundant Point e in P N (the most recent N elements) iff –e expires w.r.t P N, or – ∃ e’ s.t. e’  e, and e’ is younger than e N = 6

COMP9314Xuemin Optimality Theorem: Non-redundant Points (R N ) vs. n-of-N Skyline Query Result (Q n,N ) –(P N – R N ) does not appear in any Q n,N –Q n,N must be a subset of R N –  x  R N   n, x  Q n,N –R N = O(log d-1 N) for “independent” distributions Only need to keep R N – the minimum number of elements to be kept.

COMP9314Xuemin Querying R N critical dominance: e  e’ where e is the youngest. dominance graph G R N : R N and the critical dominance relationships.

COMP9314Xuemin Querying R N e  Q n,N iff e is a root in G R N or e’  e in G R N & e’ has expired  t(e’) < M – n + 1 <= t(e) nRNRN M-n+1Q n,N n=5{3,4,5,6,7 }3{3,4} n=4{4,5,6,7 }4{4, 7} n=3{5,6,7}5

COMP9314Xuemin Querying R N : Optimal Algorithm To answer an n-of-N Query, encode the G R N using intervals: Stab the intervals by (M-n+1). For all returned intervals (x,y), return point whose timestamp is y Technique: Use an interval tree index to achieve optimal O(log|R N |+s) query time e.g., n=6 (0,3] (0,4] (3,7] (4,5] (4,6]

COMP9314Xuemin Maintaining R N new element e new arrives: If the oldest e old  R N expires, remove e old and update R N and G R N (interval tree). find D  R N dominated by e new, update R N and G R N –Depth-first search on a R-tree of R N find e  c e new, update G R N –Best-first search on the R-tree of R N e new = 8 e old = 3 D = {6} e = 4

COMP9314Xuemin Continuous n-of-N Query Trigger-based algorithm: Deletion: Q n,N – {e old }, and Q n,N – {D} Insertion: Q n,N  {e new } if  (  e’  c e new and t(e’ ) >= M-n+1) Maintain a min-heap of Q n,N for efficiency e new = 8 M = 8 e old = 3 D = {6} e’ = 4 M = 7 n = 4, N=5 Q 4,5 = {4,7} M = 8 n = 4, N=5 Q 4,5 = {5,7,8} Q 5,5 = {3,4}Q 5,5 = {4,7}

COMP9314Xuemin (n 1,n 2 )-of-N Query More complicated than n-of-N Query P N needs to be kept! (Old) critical dominance: t (a e ) = max { t (e’): e’  e & t (e’) < t (e) } backward critical dominance: t (b e ) = min { t (e’): e’  e & t (e’) > t (e)} e  Q (n1,n2),N iff a e < M-n 2 +1  e  M-n 1 +1 < b e CBC dominance graph: P N & the two kinds of dominance relationships (2,4)-of-7: {4, 6} a 5 = 3, b 5 = 6 (M-n 2 +1, M-n 1 +1] = (4,6]

COMP9314Xuemin Processing (n 1,n 2 )-of-N Query Encode the CBC dominance graph: –e  ((a e, e], b e ) –build an interval tree on (a e, e] only Stab using M-n 2 +1 against the interval tree and check e <= M-n 1 +1 < b e ) on-the-fly: –O(logN+s*), sub-optimal (2,4)-of-7: ??? (M-n 2 +1, M-n 1 +1] = (4,6] (1,6] (3,4] (3,5]... (2,4)-of-7: {4, 6} Candidates: {4, 5, 6}

COMP9314Xuemin More on (n 1,n 2 )-of-N Query Maintenance: Similar to that of n-of-N query, but –Always expires the oldest element in P N, and maintain the interval tree and the R-tree on R N. –Implementation-wise: Use two interval trees to index R N and P N - R N, respectively. Continuous queries –More complicated A new skyline point might not be a skyline in the previous result, nor critically dominated by a skyline point in the previous result nor a newly arrived point –Basic idea Maintain additional Candidate Solutions (minimization) & triggers –Details in the full paper

COMP9314Xuemin Experiment Setup Hardware –P4 2.8G CPU, 1G Memory Datasets –Correlated, independent, and anti- correlated –d = 2 to 5, N = 10 6 Algorithms –KLP, nN, mnN, cnN, n12N, mn12N Metrics –Processing time  Streaming rate

COMP9314Xuemin n-of-N Query Varying dimensionality –M up to 2M, N = 1M, n uniformly from [1K, 1M], #queries = 1000

COMP9314Xuemin n-of-N Query (cont’d) Varying n for correlated, independent, and anti-correlated datasets

COMP9314Xuemin Maintenance Costs 2d and 5d datasets, measure average and max time, N = i * 10 5

COMP9314Xuemin Scalability M (total number) = 2M, N = 1M, #queries = 2M anti-correlatedindependent

COMP9314Xuemin Continuous n-of-N Queries 2d & 5d datasets N = 10K and 1M 10 queries with n = i*(N/10) measures cnN avg, cnN max, nN avg, nN max

COMP9314Xuemin (n 1,n 2 )-of-N Queries Varying dimensionality –M up to 2M, N = 1M, #queries = 1000 –restricting n 2 – n 1 >= 500 Scalability – M = 2M, N = 1M, #queries = 2M

COMP9314Xuemin Maintenance 2d and 5d datasets measure average and max time N = i * 10 5

COMP9314Xuemin Conclusions Efficient algorithms for various sliding windows skyline queries –Keep only minimum number of points –Encode and index those points –Maintain all the data structures The proposed solutions –have theoretical guarantee on the performance, and –have demonstrated efficiency and scalability in the experiments Future work –Improve the current solution for (n 1,n 2 )-of-N queries –Approximate skyline queries

COMP9314Xuemin Q&A Thank You!

COMP9314Xuemin Reference [ICDE01] S. Borzsonyi, D. Kossmann, and K. Stocker. The skyline operator. ICDE, [VLDB01] K. Tan, P. Eng, and B. Ooi. Efficient progressive skyline computation. VLDB, [VLDB 02] D. Kossmann, F. Ramsak, and S. Rost. Shooting stars in the sky: An online algorithm for skyline queries. VLDB, [SIGMOD03] D. Papadias, Y. Tao, G. Fu, and B. Seeger. An optimal progressive alogrithm for skyline queries. SIGMOD, [SIAM J. comp00] S. Kapoor. Dynamic maintenance of maxima of 2- d point sets. SIAM J. Comput., 2000.