1. Efficient Skyline Computation on Vertically Partitioned Datasets Dimitris Papadias, David Yang, Georgios Trimponias CSE Department, HKUST, Hong Kong

2. Outline  Problem Statement  Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm  Algorithms for Top-k Query Processing  FM Sketches  Putting Everything Together

3. Outline  Problem Statement  Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm  Algorithms for Top-k Query Processing  FM Sketches  Putting Everything Together

4. A Motivating Example  Consider a database containing information about hotels. The y-dimension represents the price of the room, whereas the x-dimension captures the distance of the room from the beach. Distance Price Skyline objects Hotel roomsp Dominance Region of p Borders of p’s Dominance Region

5. Skyline Preliminaries [ICDE, 2001]  Skylines constitute a very useful tool in numerous disciplines, such as for multidimensional decision making and data mining.  Given a set of d-dimensional objects p 1, …, p N, the skyline operator retrieves all these objects that are nor dominated by any other object in the set.  An object p i dominates another point p j, if it is not worse than p j in all dimensions and better than it in at least one dimension.  Properties: The top-1 tuple according to any preference function that assigns scores to tuples is in the skyline tuple. Conversely, for any skyline tuple, there exists a preference function according to which it is the top-1.

6. 4 Common Data Distributions

7. Problem Definition  Compute the skyline when the dataset is vertically decomposed among a set of N servers.  Goal: minimize the data that must be retrieved from each server.  We assume wireless environments, where communication overhead constitutes the dominant factor in battery consumption.  Consider mobile phone applications as real world examples.

8. (a) Subspace D 1 at server N 1 (b) Subspace D 2 at Server N 2 First Observations  The global skyline may contain points that do not appear in the local skylines.  Instead of transmitting all records over the network, avoid sending out points that are guaranteed to be dominated globally by an anchor point.

9. Outline  Problem Statement  Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm  Algorithms for Top-k Query Processing  FM Sketches  Putting Everything Together

10. Balke’s Algorithm [EDBT, 2004]  Assume that the d-dimensional database is vertically partitioned into d lists, one for each dimension, assigned to different servers. The lists contain values in ascending order.  Idea: perform sorted accesses on the d lists in a round- robin manner, until a point p (anchor), is reached in every list.  Points that have not showed up at this moment in any list can be safely pruned, since they are dominated by the anchor.

11. Example  Let a 2-dimensional database with the following two lists:  L1  L2 Pointabdmgc Value123556 Pointcdekab Value123457 … …

12. Example (cont.)  Let a 2-dimensional database with the following two lists:  L1  L2 Pointabdmgc Value123556 Pointcdekab Value123457 … … The first point to be retrieved from both lists.

13. Example (cont.)  Let a 2-dimensional database with the following two lists:  L1  L2 Pointabdmgc Value123556 Pointcdekab Value123457 … … The first point to be retrieved from both lists.These points cannot be part of the skyline.

14. Further Improvement  Efficiency can be improved, if instead of visiting the lists in a round-robin manner, we access the most promising list with random accesses.  As a result, only the least expansion is performed on each list. ∙ ∙ ∙ P ∙ ∙ ∙ L1 ∙ ∙ ∙ P ∙ ∙ ∙ L2 avoid visiting these points

15. Outline  Problem Statement  Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm  Algorithms for Top-k Query Processing  FM Sketches  Putting Everything Together

16. Setting  Let N 1,.., N m be m servers storing the same dataset DB.  For each record P  DB every server N i maintains a local score s i (P), and sorts all records in decreasing order of their local scores.  A client wishes to obtain the k records of DB with the maximum global score s.  The score is computed using a monotonic function f on the local scores, i.e., s(P) = f(s 1 (P),.., s m (P)).  Goal: minimize the required number of accesses.

17. Fagin’s Algorithm [PODS, 2001]  Each server N i performs sorted round-robin accesses and sends to the client the next record and its local score.  When the first common record P anc is encountered by all servers, the client terminates the sorted accesses.  Then, it obtains the missing local scores of the other encountered points through random accesses.  The candidate with the highest global score is the top-1 result.

18. Threshold Algorithm [PODS, 2001]  It utilizes an upper bound  TA on the global score to terminate earlier than FA.  The client retrieves the local scores of newly encountered points with random accesses at the remaining servers and computes their global scores, and picks the best score s best.  The threshold  TA is equal to the sum of the local thresholds at each server.  As long  TA > s best, TA continues the sorted accesses, while it keeps updating  TA.  Eventually, the top-1 point will be returned.

19. Example for FA and TA

20. Best Position Algorithm [VLDB, 2007]  It further improves TA by utilizing a tighter threshold.  Let bp i be the position at server N i such that all points up to bp i have been encountered through sorted or random accesses.  The global threshold  BP is equal to the sum of the local thresholds at bp i.

21. Outline  Problem Statement  Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm  Algorithms for Top-k Query Processing  FM Sketches  Putting Everything Together

22. Flajolet / Martin sketches [JCSS ’85]  Goal: Estimate the distinct number of objects from a small-space representation of a set.  Sketch of a union of items is the OR of their sketches  Insertion order and duplicates don’t matter! Prerequisite: Let h be a random, binary hash function. Sketch of an item For each unique item with ID x, For each integer 1 ≤ i ≤ k in turn, Compute h (x, i). Stop when h (x, i) = 1, and set bit i. X 00100 Z 10000 X Z 10100 ∩

23. Flajolet / Martin sketches (cont.) Estimating COUNT Take the sketch of a set of N items. Let j be the position of the leftmost zero in the sketch. j is an estimator of log 2 (0.77 N) Fixable drawbacks: Estimate has faint bias Variance in the estimate is large. 110 1 S 1 Best guess: COUNT ~ 11 j = 3

24. Flajolet / Martin sketches (cont.)  Standard variance reduction methods apply.  Compute m independent sketches in parallel.  Compute m independent estimates of N.  Take the mean of the estimates.  Provable tradeoffs between m and variance of the estimator.

25. Application to COUNT in Sensor Databases Each sensor computes k independent sketches of itself (using unique ID x) –sensor computes a sketch of its value. Use a robust routing algorithm to route sketches up to the sink. Aggregate the k sketches via union en-route. (OR) The sink then estimates the count. sink S1S1 S3S3 S2S2 S4S4 S1S1 S2S2 S1∪S2∪S3S1∪S2∪S3 S4S4 S1∪S2∪S3∪S3S1∪S2∪S3∪S3

26. Outline  Problem Statement  Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm  Algorithms for Top-k Query Processing  FM Sketches  Putting Everything Together

27. Problem Characteristics  Each vertical decomposition has arbitrary dimensionality, contrary to Balke’s setting.  Anchor selection substantially determines the total number of transmitted data.  VPS adopts sorting on the local dominance. In particular, the local dominance count dom i (P) of a point P with respect to subspace D i is the number of points dominated by P in D i.  Balke selects as the anchor, the data point P with the maximal dom SUM (P).  We utilize a tighter upper bound for dom(P) is the minimum dom MIN among all local dominance counts.

28. Anchor Selection (a) Subspace D 1 at server N 1 (b) Subspace D 2 at Server N 2 C: optimal anchor point A: has maximal dom MIN B: has maximal dom SUM

29. Our algorithm on the previous example

30. 1 st Optimization: Multiple Anchor Points  The previous algorithm performs pruning with a single anchor P anc. Specifically, a point P that is locally dominated by P anc in all subspaces is not sent to the client.  On the other hand, if P is incomparable with P anc even in a single subspace D i, it will be transmitted by the corresponding server N i.  We suggest that multiple points can often achieve more effective pruning.

31. Pruning with 2 points

32. 2 nd Optimization: Integration of Sketches  So far, we have estimated the (expected) global dominance dom(P) of a point P using dom MIN (P).  This approach is biased towards points that have high local dominance counts in all subspaces, but dominate few records globally (A).  Thus, we propose an unbiased approach that directly estimates the global dominance counts using sketches that count the number of distinct objects approximately.  We assume that each N i server has a local dominance sketch sk i (P) for every point P, which aggregates all points that P dominates locally in D i.

33. Experiments

34. Thank you!