1 Lightweight Indexing of Observational Data in Log-Structured Storage National University of Singapore (Sheng Wang, Beng Chin Ooi) Portland State University(David Maier) VLDB 2014

2 Outline  Background  Challenges  Contributions  CR-Index  Index Optimization  Experiments

3 background Huge amounts of data generated by sensors every day Data are expanding in precision and quantity High write throughput Efficient query

4 challenges State-of-the-art storage doesn't take high-throughput into account (CMOP : RDBMS + netCDF) Record-level Index incur significant index maintenance cost –B + -Tree Random IO due to update –LSM-Tree large number of index entries

5 contributions A schema for storing observational data in logBase [1] to facilitate indexing A novel, lightweight index called CR-Index structure for range queries which take full advantage of observational-data traits Experiments on two real-word observational datasets [1] H. T. Vo, S. Wang, D. Agrawal, G. Chen, B. C. Ooi. Int'l Conference on Very Large Data Bases (VLDB), PVLDB 5(10): , 2012.

6 Storage High write throughput Traits of observational data –no update –continuous change –potential discontinuities Log-structured storage append new data to the end of a file avoid random I/O

7 LogBase An unordered column-oriented distributed log-store Each node is responsible for one or more partitions of a table Version controll and transaction semantics Relational data model –each record has a primary key and several attributes –each record is decomposed as a set of cells (KEY, ATTRIBUTE, VALUE, TIMESTAMP)

8 Architecture

9 Logic View & Physical View

10 Basic query formats Time range Value range

11 Obervational data locality In general, append only strategy hurts read performance, but log-store provide considerable data locality –time-ordered property time range query : sequencial scan –value-correlated property due to continuity trait, once a record is inside a value range, surrounding records will likely lie in the range

12 Continuous Range Index(CR- Index) the value-correlated property implies a seek in the log can potentially yield many results we needn't locate qualifying records individualy, as long as identify regions containing results Group successive records into blocks as an atomic unit Each block is summarized by a value range using a boundary pair Range query can be transformed to an intersection- checking problem

13 CR-Index structure

14 CR-Index structure

15 Index structures Interval tree –O(n) –stabbing query:O(logn)

16 Solution partition result set into two disjoint group –Group A : CR-Record that have at least one endpoint inside the query range [a,b] B + -tree –Group B : CR-Records that completely contain the query range Interval-tree

17 Solution For each CR-record, two entries are inserted into the B + -tree, one for each endpoint. the endpoint as a key and CR- recrod's refrence is the value For each CR-record, its' value range is inserted into the Interval tree

18 Example

19 Example

20 Example

21 Query example query1 : [3,11] result1 : 2,3,4,5,3,2,4 2,3,4,5 query2 : [16,18] result2 :

22 Query example query : [16,18] result : 6

23 Optimization Index with delta intervals –boundary pair in consective blocks may overlap, if a query intersect a block, it will probably intersect the following blocks

24 Example query : [3,5] result : 2, 3

25 Length-k delta interval

26 Evaluate range query value condition can be used at both interval-index level and CR-log level time condition can be used at CR-Log via checkpoint-list hole information can be updated

27 Query steps 1.Access the interval index to get CR-Record ids: Group A from B+-tree and Group B from interval tree 2.Locate each identified record in the CR-Log. Scan the log for additional CR-records if using delta intervals 3.Filter CR-record using checkpoint list and hole information 4.Fetch and scan the data blocks for remaining CR- records. Exract and return all qualifying results 5.For any detected false-positive blocks, track the holes and update the hole information in CR-records.

28 Experiments Compare with B + -tree and LSM-tree Data sets –CMOP Costal Margin Data (13million) salinity, temperature, oxygen –Real-time Soccer Game Date (25million) sensor ID, position, speed, velocity, accelaeration

29 Experiment Environment –A cluster where each machine has a quad- core processor, 8GB, 500GB –Java –block length : 64 –delta-interval : 1 –range query

30 System load time Write time in load data CR-Index : 8% LSM : 45%-77% B+ : 78%-124%

31 Index update time 15% LSM-tree 9% B+-tree

32 Index space consumption disk space –10%-12% LSM-tree –4%-6% B+-tree

33 Query response time

34 Index look-up cost

35 Data access cost

36 Over QQQ