Spatial Online Sampling and Aggregation

Spatial Online Sampling and Aggregation
Lu Wang, Ke Yi Hong Kong University Of Science and Technology Robert Christensen, Feifei Li University of Utah

Motivation Geo Spatial Data is being collected on a massive scale Approximate aggregations is fast and often effective for this data Users are interested in interactive data analysis * Cell phone trajectories. * Weather data. * Wearables Approximate answers can be sufficient because it supports interactive queries. Exact answers can always be collected offline Spatial Online Sampling and Aggregation

Problem description Uniform sampling of spatial-temporal points for a query region 𝑄 Samples from query are reported online We are focused on reporting samples Aggregation estimates can be calculated using traditional statistical methods. Samples can be an input to other data processing algorithms Spatial Online Sampling and Aggregation

Running Example b k d Spatial data points j c g l i o e Point labels h m a f n Query region 𝑄 Spatial Online Sampling and Aggregation

Baseline Methods Query First Calculate 𝑆=𝑃∩𝑄. Extract samples from 𝑆 upon request Must materialize all potential samples At least as costly as reporting all data in 𝑄 Expensive when 𝑄 is large b k d j c g l i o e h m a f n Results Buffer e f g h i Spatial Online Sampling and Aggregation

Baseline Methods Query First Sample First Upon request, pick 𝑝∈𝑃, if 𝑝∈𝑄 report, otherwise reject and repeat Probability of rejection = 1− 𝑄 |𝑃| Costly if 𝑄 is small. b k d j c g l i o e h m a Optimizations to Query Frist and Sample First have been implemented. Descriptions of the optimizations are found in the paper. The effects of the optimizations are described in the experimental results. f n Spatial Online Sampling and Aggregation

LS-Tree We propose the Level Sampling Tree (LS-Tree) LS-Tree is a collection of several R-trees of different sizes The largest R-tree contains all data points The smallest R-tree contains a very small sample of the data points Queries involve performing Query First operations on progressively larger R-trees Spatial Online Sampling and Aggregation

LS-Tree Building Level Sample Tree Build R-Tree over 𝑆 0 =𝑃 Continue to build R-Tree over 𝑆 𝑖+1 by sampling from 𝑆 𝑖 with probability 1/2 End when 𝑆 ℓ is sufficiently small 𝑆 0 𝑆 1 𝑆 2 a b d e f g h i c j k l m n o a d f g j m n f j m Spatial Online Sampling and Aggregation

LS-Tree Queries using LS-Tree Use Query First on 𝑆 ℓ . After samples are exhausted, do Query First on 𝑆 𝑖−1 Only return samples not previously reported. 𝑆 0 𝑆 1 𝑆 2 a b d e f g h i c j k l m n j m b k d j d j c LS-tree must pause to perform Query First queries, causing periodic delays in the online sampling. As the query progresses, state must be kept to remember which samples have been previously reported. g l g i o e h m m a a f f f n n Spatial Online Sampling and Aggregation

LS-Tree LS-tree starts to report samples quickly Samples are reported without replacement Verifying without replacement becomes more costly as the query continuously asks for more and more samples Requires maintenance of multiple R-trees Expensive when supporting transactions LS-tree must pause to perform Query First queries, causing periodic delays in the online sampling. As the query progresses, state must be kept to remember which samples have been previously reported. Spatial Online Sampling and Aggregation

RS-tree: Modifying the structure of internal nodes
Add a buffer of samples of points from subtree rooted at the node 𝑢. Each 𝑝∈𝑃 𝑢 has equal probability of being an entry in the sample buffer at 𝑢 Spatial Online Sampling and Aggregation

RS-Tree Filling the sample buffers in RS-tree Recursive algorithm, randomly decide how many samples to request from each child node. Sample buffers unneeded in leaf nodes. Root (𝑑𝑒𝑝𝑡ℎ=0) 𝑑𝑒𝑝𝑡ℎ=1 Leaves (𝑑𝑒𝑝𝑡ℎ=2) a b d e f g h i c j k l m n a b d e f g h i c j k l m n o a b d e f g h i c j k l m n o samples b k d j c MBB of given node g l i o e h m a f n Spatial Online Sampling and Aggregation

RS-Tree Queries over RS-tree Start at the root node. Scan the sample buffer, report those within 𝑄 Descend the tree, only add nodes which intersect 𝑄 Obtain samples from nodes with appropriate probability during sampling Root (𝑑𝑒𝑝𝑡ℎ=0) 𝑑𝑒𝑝𝑡ℎ=1 Leaves (𝑑𝑒𝑝𝑡ℎ=2) a b d e f g h i c j k l m n a b d e f g h i c j k l m n a b d e f g h i c j k l m n o a b d e f g h i c j k l m n o Lets consider this instance in more detail o Spatial Online Sampling and Aggregation

RS-Tree We do not know how many points inside a node satisfy the query We sample each node with probability proportional to number of total items in node. If 𝑝∈𝑄 accept, otherwise reject and repeat b k d j c g l i o e h m a f n w.p. 𝟑 𝟕 w.p. 𝟒 𝟕 Spatial Online Sampling and Aggregation

RS-Tree An estimation of query size is important for aggregations such as SUM and COUNT Estimating query size Use frontier list of a query cursor If 𝑅 𝑢 ⊆𝑄, all 𝑃 𝑢 elements inside 𝑄 Otherwise estimate 𝑃 𝑢 ∩𝑄 using the uniform samples of the node |𝑃 𝑢 | 𝑠𝑎𝑚𝑝𝑙𝑒𝑠 𝑢 ∩𝑄 𝑠𝑎𝑚𝑝𝑙𝑒𝑠 𝑢 Query size is a basic value needed for many aggregation queries (SUM, count) Proof and calculations of standard deviation in the paper Spatial Online Sampling and Aggregation

RS-Tree Queries avoid disk access Samples start being reported quickly Minimal added overhead (verses R-tree) Algorithms for efficient modifications to the RS-tree Samples reported with replacement Spatial Online Sampling and Aggregation

Experimental Setup Open Street Maps data (2.2 billion points) RS-tree: Range Sampling Tree implementation LS-tree: Level Sampling Tree implementation Random Path: Sample First over R-tree Range Report: Query First implementation Random Shuffle: Sample First with linear scan 𝑞= number of items inside query region 𝑄 𝑘= number of samples requested for query We will review a few experiments. A complete experimental section exists in the paper. Spatial Online Sampling and Aggregation

Analysis of each sampling method (Time & I/O)
Spatial Online Sampling and Aggregation

Cost of Building LS-tree must build multiple R-trees, taking additional time and space. RS-tree builds an R-tree and fills sample buffer for each inner node. Index Size Raw data 50 GB R-tree 75 GB RS-tree 80 GB LS-tree 151 GB Spatial Online Sampling and Aggregation

Aggregation query LS-tree starts reporting faster then RS-tree RS-tree converges faster then LS-tree Exact answer is about 1800 Spatial Online Sampling and Aggregation

Query speed over large data (2.2 billion points)
(vary sampling percentage) (𝒒=2.2 million) (vary number of points in full query) (𝒌=10,000) Spatial Online Sampling and Aggregation

RS-tree in real world application
RS-tree is used to support fast spatial samples in the project STORM: Spatio-Temporal Online Reasoning and Management Best Demo Award - SIGMOD 2015 Spatial Online Sampling and Aggregation

Conclusion Developed spatial indexes supporting sampling LS-tree Stores multiple levels of samples in a collection of R-tree indexes RS-tree Caches samples in each R-tree node. Performed an in depth analysis of each sampling technique Experimentally validated the superior performance of LS-tree and RS-tree verse baseline methods. Spatial Online Sampling and Aggregation

Conclusion Thank You Spatial Online Sampling and Aggregation

RS-tree time to estimate size
Random regions of various sizes are queried. Using 10% of the time to count the size exactly in R-tree, we estimate the size and calculate the error Spatial Online Sampling and Aggregation

RS-tree digram Spatial Online Sampling and Aggregation

Vary sample buffer size
Small data set (24 million points) Query returns 400k samples Spatial Online Sampling and Aggregation

Aggregation query Exact answer is about 1800 LS-tree starts reporting samples most quickly, but converges slowly RS-tree reports samples faster once they are being reported. LS-tree starts reporting faster then RS-tree RS-tree converges faster then LS-tree Exact answer is about 1800 Spatial Online Sampling and Aggregation

Analysis of each sampling method (Time & I/O)
Spatial Online Sampling and Aggregation

Hoeffding’s inequality
Let X1, ..., Xn be independent random variables bounded by the interval [0, 1], i.e., 0 ≤ Xi ≤ 1. We define the empirical mean of these variables by: X=1/n( X1+ X2+…+Xn) Then: If we know ai<Xi<bi

Central limit theorem Let {X1, ..., Xn} be a random sample of size n— that is, a sequence of independent and identically distributed (iid) random variables drawn from distributions of expected values given by µ and finite variances given by σ2. Suppose we are interested in the sample average

Spatial Online Sampling and Aggregation

Similar presentations

Presentation on theme: "Spatial Online Sampling and Aggregation"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Spatial Online Sampling and Aggregation

Similar presentations

Presentation on theme: "Spatial Online Sampling and Aggregation"— Presentation transcript:

Similar presentations

About project

Feedback