Presentation is loading. Please wait.

Presentation is loading. Please wait.

Distributed Query Processing. Agenda Recap of query optimization Transformation rules for P&D systems Memoization Queries in heterogeneous systems Query.

Similar presentations


Presentation on theme: "Distributed Query Processing. Agenda Recap of query optimization Transformation rules for P&D systems Memoization Queries in heterogeneous systems Query."— Presentation transcript:

1 Distributed Query Processing

2 Agenda Recap of query optimization Transformation rules for P&D systems Memoization Queries in heterogeneous systems Query evaluation strategies Eddies

3 Introduction Alternative ways of evaluating a given query –Equivalent expressions –Different algorithms for each operation (Chapter 13) Cost difference between a good and a bad way of evaluating a query can be enormous –Example: performing a r X s followed by a selection r.A = s.B is much slower than performing a join on the same condition Need to estimate the cost of operations –Depends critically on statistical information about relations which the database must maintain –Need to estimate statistics for intermediate results to compute cost of complex expressions

4 Introduction (Cont.) Relations generated by two equivalent expressions have the same set of attributes and contain the same set of tuples, although their attributes may be ordered differently.

5 Introduction (Cont.) Generation of query-evaluation plans for an expression involves several steps: 1.Generating logically equivalent expressions Use equivalence rules to transform an expression into an equivalent one. 2.Annotating resultant expressions to get alternative query plans 3.Choosing the cheapest plan based on estimated cost The overall process is called cost based optimization.

6 Equivalence Rules 1.Conjunctive selection operations can be deconstructed into a sequence of individual selections. 2.Selection operations are commutative. 3.Only the last in a sequence of projection operations is needed, the others can be omitted. 4.Selections can be combined with Cartesian products and theta joins. a.   (E 1 X E 2 ) = E 1  E 2 b.   1 (E 1  2 E 2 ) = E 1  1   2 E 2

7 Equivalence Rules (Cont.) 5.Theta-join operations (and natural joins) are commutative. E 1  E 2 = E 2  E 1 6.(a) Natural join operations are associative: (E 1 E 2 ) E 3 = E 1 (E 2 E 3 ) (b) Theta joins are associative in the following manner: (E 1  1 E 2 )  2   3 E 3 = E 1  2   3 (E 2  2 E 3 ) where  2 involves attributes from only E 2 and E 3.

8 Pictorial Depiction of Equivalence Rules

9 Equivalence Rules (Cont.) 7.The selection operation distributes over the theta join operation under the following two conditions: (a) When all the attributes in  0 involve only the attributes of one of the expressions (E 1 ) being joined.   0  E 1  E 2 ) = (   0 (E 1 ))  E 2 (b) When  1 involves only the attributes of E 1 and  2 involves only the attributes of E 2.   1    E 1  E 2 ) = (   1 (E 1 ))  (   (E 2 ))

10 Equivalence Rules (Cont.) 8.The projections operation distributes over the theta join operation as follows: (a) if  involves only attributes from L 1  L 2 : (b) Consider a join E 1  E 2. – Let L 1 and L 2 be sets of attributes from E 1 and E 2, respectively. –Let L 3 be attributes of E 1 that are involved in join condition , but are not in L 1  L 2, and – let L 4 be attributes of E 2 that are involved in join condition , but are not in L 1  L 2.

11 Equivalence Rules (Cont.) 9.The set operations union and intersection are commutative E 1  E 2 = E 2  E 1 E 1  E 2 = E 2  E 1 n(set difference is not commutative). 10.Set union and intersection are associative. (E 1  E 2 )  E 3 = E 1  (E 2  E 3 ) (E 1  E 2 )  E 3 = E 1  (E 2  E 3 ) 11.The selection operation distributes over ,  and –.   (E 1 – E 2 ) =   (E 1 ) –   (E 2 ) and similarly for  and  in place of – Also:   (E 1 – E 2 ) =   (E 1 ) – E 2 and similarly for  in place of –, but not for  12.The projection operation distributes over union  L (E 1  E 2 ) = (  L (E 1 ))  (  L (E 2 ))

12 Multiple Transformations (Cont.)

13 Optimizer strategies Heuristic –Apply the transformation rules in a specific order such that the cost converges to a minimum Cost based –Simulated annealing –Randomized generation of candidate QEP –Problem, how to guarantee randomness

14 Memoization Techniques How to generate alternative Query Evaluation Plans? –Early generation systems centred around a tree representation of the plan –Hardwired tree rewriting rules are deployed to enumerate part of the space of possible QEP –For each alternative the total cost is determined –The best (alternatives) are retained for execution –Problems: very large space to explore, duplicate plans, local maxima, expensive query cost evaluation. –SQL Server optimizer contains about 300 rules to be deployed.

15 Memoization Techniques How to generate alternative Query Evaluation Plans? –Keep a memo of partial QEPs and their cost. –Use the heuristic rules to generate alternatives to built more complex QEPs –r 1 r 2 r 3 r 4 r 1 r 2 r 2 r 3 r 3 r 4 r 1 r 4 x Level 1 plans r3 r3 r3 r3 Level 2 plans Level n plans r4 r4 r 2 r 1

16 Distributed Query Processing For centralized systems, the primary criterion for measuring the cost of a particular strategy is the number of disk accesses. In a distributed system, other issues must be taken into account: –The cost of a data transmission over the network. –The potential gain in performance from having several sites process parts of the query in parallel.

17 Par &dist Query processing The world of parallel and distributed query optimization –Parallel world, invent parallel versions of well-known algorithms, mostly based on broadcasting tuples and dataflow driven computations –Distributed world, use plan modification and coarse grain processing, exchange large chunks

18 Transformation rules for distributed systems Primary horizontally fragmented table: –Rule 9: The union is commutative E 1  E 2 = E 2  E 1 –Rule 10: Set union is associative. (E 1  E 2 )  E 3 = E 1  (E 2  E 3 ) –Rule 12: The projection operation distributes over union  L (E 1  E 2 ) = (  L (E 1 ))  (  L (E 2 )) Derived horizontally fragmented table: –The join through foreign-key dependency is already reflected in the fragmentation criteria

19 Transformation rules for distributed systems Vertical fragmented tables: –Rules: Hint look at projection rules

20 Optimization in Par & Distr Cost model is changed!!! –Network transport is a dominant cost factor The facilities for query processing are not homogenous distributed –Light-resource systems form a bottleneck –Need for dynamic load scheduling

21 Simple Distributed Join Processing Consider the following relational algebra expression in which the three relations are neither replicated nor fragmented account depositor branch account is stored at site S 1 depositor at S 2 branch at S 3 For a query issued at site S I, the system needs to produce the result at site S I

22 Possible Query Processing Strategies Ship copies of all three relations to site S I and choose a strategy for processing the entire locally at site S I. Ship a copy of the account relation to site S 2 and compute temp 1 = account depositor at S 2. Ship temp 1 from S 2 to S 3, and compute temp 2 = temp 1 branch at S 3. Ship the result temp 2 to S I. Devise similar strategies, exchanging the roles S 1, S 2, S 3 Must consider following factors: –amount of data being shipped –cost of transmitting a data block between sites –relative processing speed at each site

23 Semijoin Strategy Let r 1 be a relation with schema R 1 stores at site S 1 Let r 2 be a relation with schema R 2 stores at site S 2 Evaluate the expression r 1 r 2 and obtain the result at S 1. 1. Compute temp 1   R1  R2 (r1) at S1. 2. Ship temp 1 from S 1 to S 2. 3. Compute temp 2  r 2 temp1 at S 2 4. Ship temp 2 from S 2 to S 1. 5. Compute r 1 temp 2 at S 1. This is the same as r 1 r 2.

24 Formal Definition The semijoin of r 1 with r 2, is denoted by: r 1 r 2 it is defined by:  R1 (r 1 r 2 ) Thus, r 1 r 2 selects those tuples of r 1 that contributed to r 1 r 2. In step 3 above, temp 2 =r 2 r 1. For joins of several relations, the above strategy can be extended to a series of semijoin steps.

25 Join Strategies that Exploit Parallelism Consider r 1 r 2 r 3 r 4 where relation ri is stored at site S i. The result must be presented at site S 1. r 1 is shipped to S 2 and r 1 r 2 is computed at S 2 : simultaneously r 3 is shipped to S 4 and r 3 r 4 is computed at S 4 S 2 ships tuples of (r 1 r 2 ) to S 1 as they produced; S 4 ships tuples of (r 3 r 4 ) to S 1 Once tuples of (r 1 r 2 ) and (r 3 r 4 ) arrive at S 1 (r 1 r 2 ) (r 3 r 4 ) is computed in parallel with the computation of (r 1 r 2 ) at S 2 and the computation of (r 3 r 4 ) at S 4.

26 Query plan generation Apers-Aho-Hopcroft –Hill-climber, repeatedly split the multi-join query in fragments and optimize its subqueries independently Apply centralized algorithms and rely on cost-model to avoid expensive query execution plans.

27 Query evaluators

28 Query evaluation strategy Pipe-line query evaluation strategy –Evaluation: Oriented towards OLTP applications –Granule size of data interchange Items produced one at a time No temporary files –Choice of intermediate buffer size allocations Query executed as one process Generic interface, sufficient to add the iterator primitives for the new containers. CPU intensive Amenable to parallelization

29 Query evaluation strategy Pipe-line query evaluation strategy –Called Volcano query processing model –Standard in commercial systems and MySQL Basic algorithm: –Demand-driven evaluation of query tree. –Operators exchange data in units such as records –Each operator supports the following interfaces:– open, next, close open() at top of tree results in cascade of opens down the tree. An operator getting a next() call may recursively make next() calls from within to produce its next answer. close() at top of tree results in cascade of close down the tree

30 Volcano Refresher Query SELECT name, salary*.19 AS tax FROM employee WHERE age > 25 Try to maximize performance

31 Volcano Refresher Operators Iterator interface -open() -next(): tuple -close() Try to maximize performance

32 The Volcano model is based on a simple pull-based iterator model for programming relational operators. The Volcano model minimizes the amount of intermediate store The Volcano model is CPU intensive and inefficient Try to maximize performance Volcano paradigm

33 MonetDB paradigm The MonetDB kernel is a programmable relational algebra machine Relational operators operate on ‘array’-like structures Try to use simple a software pattern

34 Query evaluation strategy Materialized evaluation strategy –Used in MonetDB –Basic algorithm: for each relational operator produce the complete intermediate result using materialized operands –Evaluation: Oriented towards decision support queries Limited internal administration and dependencies Basis for multi-query optimization strategy Memory intensive Amendable for distributed/parallel processing

35 SQL MonetDB Server MonetDB Kernel XQuery MAL function user.s3_1():void; X1:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",0); X6:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",1); X9:bat[:oid,:lng] := sql.bind("sys","photoobjall","objid",2); X13:bat[:oid,:oid] := sql.bind_dbat("sys","photoobjall",1); X8 := algebra.kunion(X1,X6); X11 := algebra.kdifference(X8,X9); X12 := algebra.kunion(X11,X9); X14 := bat.reverse(X13); X15 := algebra.kdifference(X12,X14); X16 := calc.oid(0@0); X18 := algebra.markT(X15,X16); X19 := bat.reverse(X18); X20 := aggr.count(X19); sql.exportValue(1,"sys.","count_","int",32,0,6,X20,""); end s3_1; select count(*) from photoobjall; Try to use simple a software pattern

36 Operator implementation All algebraic operators materialize their result Local optimization decisions Heavy use of code expansion to reduce cost –55 selection routines –149 unary operations –335 join/group operations –134 multi-join operations –72 aggregate operations Try to use simple a software pattern

37 Micro-benchmark MonetDB/SQL0.34 N44 MySQL25.1 N238 PostgreSQL10.6 N1230 Commercial 139.0 N800 Commercial 217 N150 In milliseconds/10K Fixed cost in ms Keeping the query result in a new table is often too expensive select * into tmp from tapestry where attr1>=0 and attr1 <=@range create table tmp( attr0 int, attr1 int); insert into tmp select * from tapestry where attr1>=0 and attr1 <=@range ;

38 Multi-column tapestry Experiments ran on Athlon 1.4, Linux commercial MonetDB/SQL #joins ms

39 A column store should be designed from scratch to benefit from its characteristics Simulation of a column store on top of an n-ary system using the Volcano model does not work

40 Try to maximize performance Paste Present Potency Execution Paradigm Database Structures Query optimizer

41 Applications have different characteristics Platforms have different characteristics The actual state of computation is crucial A generic all-encompassing optimizer cost-model does not work Try to avoid the search space trap

42 SQL MonetDB Server MonetDB Kernel XQuery MAL Operational optimizer: – Exploit everything you know at runtime – Re-organize if necessary Try to disambiguate decisions

43 SQL MonetDB Server MonetDB Kernel XQuery MAL Strategic optimizer: – Exploit the semantics of the language – Rely on heuristics Operational optimizer: – Exploit everything you know at runtime – Re-organize if necessary Try to disambiguate decisions

44 SQL MonetDB Server Tactical Optimizer MonetDB Kernel XQuery MAL y1:bat[:oid,:dbl]:= bpm.take("sys_photoobjall_ra"); y2 := bpm.new(:oid,:oid); barrier rs:= bpm.newIterator(y1,A0,A1); t1:= algebra.uselect(rs,A0,A1); bpm.addSegment(y2,t1); redo rs:= bpm.hasMoreElements(y1,A0,A1); exit rs; x1:bat[:oid,:dbl]:= sql.bind("sys","photoobjall","ra",0); x14:= algebra.uselect(x1,A0,A1); Tactical MAL optimizer: – No changes in front-ends and no direct human guidance – Minimal changes in the engine Try to disambiguate decisions

45 Code Inliner. Constant Expression Evaluator. Accumulator Evaluations. Strength Reduction. Common Term Optimizer. Join Path Optimizer. Ranges Propagation. Operator Cost Reduction. Foreign Key handling. Aggregate Groups. Code Parallizer. Replication Manager. Result Recycler. MAL Compiler. Dynamic Query Scheduler. Memo-based Execution. Vector Execution. Alias Removal. Dead Code Removal. Garbage Collector. Try to disambiguate decisions

46 Try to maximize performance Paste Present Potency Execution Paradigm Database Structures Query optimizer

47 Execution paradigms The MonetDB kernel is set up to accommodate different execution engines The MonetDB assembler program is –Interpreted in the order presented –Interpreted in a dataflow driven manner –Compiled into a C program –Vectorised processing X100 project No data from persistent store to the memory trash

48 MonetDB/x100 Combine Volcano model with vector processing. All vectors together should fit the CPU cache Vectors are compressed Optimizer should tune this, given the query characteristics. ColumnBM (buffer manager) X100 query engine CPU cache networked ColumnBM-s RAM

49 Varying the vector size on TPC-H query 1 mysql, oracle, db2 X100 MonetDB low IPC, overhead RAM bandwidth bound No data from persistent store to the memory trash

50 Vectorized-Volcano processing can be used for both multi- core and distributed processing The architecture and the parameters are influenced heavily by –Hardware characteristics –Data distribution to compress columns No data from persistent store to the memory trash

51 Does MonetDB stand a ‘real’ test? Is the main memory orientation a bottleneck? Is it functionally complete? The proof of the pudding is in the eating

52 TPC-H ATHLON X2 3800+ (2000mhz) 2 disks in raid 0, 2G main memory TPC-H 60K rows line_item table Comfortably fit in memory Performance in milliseconds

53 TPC-H ATHLON X2 3800+ (2000mhz) 2 disks in raid 0, 2G main memory Scale-factor 1 6M row line-item table Out of the box performance Queries produce empty or erroneous results

54 TPC-H ATHLON X2 3800+ (2000mhz) 2 disks in raid 0, 2G main memory

55 TPC-H ATHLON X2 3800+ (2000mhz) 2 disks in raid 0, 2G main memory

56 Code base for MonetDB/SQL is 1.2M lines of C Nightly regression testing on 17 platforms

57 Try to maximize performance Paste Present Potency Cracking B-tree, Hash Indices Materialized Views

58 Indices in database systems focus on: –All tuples are equally important for fast retrieval –There are ample resources to maintain indices MonetDB cracks the database into pieces based on actual query load Find a trusted fortune teller

59 Cracking algorithms Physical reorganization happens per column based on selection predicates. Split a piece of a column in two new pieces A<10 A>=10 A<10

60 Cracking algorithms Physical reorganization happens per column Split a piece of a column in two new pieces Split a piece of a column in three new pieces A<10 A>=10 A<10 5<A<10 A>=10 5<A<10 A<5

61 Cracking example 3 8 6 2 12 13 4 17 15 select A>5 and A<10

62 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12

63 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10

64 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10

65 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10 <=5

66 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10 <=5

67 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10 <=5

68 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10 <=5

69 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10

70 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >=10

71 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <=5

72 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <=5

73 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <=5 >5 and <10

74 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 62 15 13 4 17 12 <=5 >5 and <10

75 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 62 15 13 4 17 12 <=5 >5 and <10

76 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <=5 >5 and <10

77 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 >5 and <10

78 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 13 4 17 12 <= 5 >= 10 > 5 15

79 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries

80 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14

81 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5

82 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5

83 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5

84 racking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 >3 and <14 <=3

85 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 >3 and <14 <=3

86 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 >3 and <14 <=3

87 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 >3 and <14 <=3

88 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 <=3

89 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 10 > 5 <=3

90 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 10 > 5 <=3

91 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 10 > 5 <=3

92 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 10 > 5 <=3

93 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 10 > 5 <=3

94 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 10 > 5 <=3

95 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 10 > 5 <=3

96 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 10 > 5 <=3

97 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 14 > 5 <=3 >=10

98 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 > 3 >= 14 > 5 <=3 >=10

99 Cracking example 3 8 6 2 15 13 4 17 12 select A>5 and A<10 3 8 6 2 15 13 4 17 12 <= 5 >= 10 > 5 Improve data access for future queries select A>3 and A<14 3 8 6 2 15 13 4 17 12 >3 >= 14 > 5 <=3 >=10 The more we crack the more we learn

100 Design The first time a range query is posed on an attribute A, a cracking DBMS makes a copy of column A, called the cracker column of A A cracker column is continuously physically reorganized based on queries that need to touch attribute such as the result is in a contiguous space For each cracker column, there is a cracker index Cracker Index Cracker Column

101 A simple range query Try to avoid useless investments

102 TPC-H query 6 Try to avoid useless investments

103 Cracking is easy in a column store and is part of the critical execution path Cracking works under high volume updates Try to avoid useless investments

104 Updates Base columns are updated as normally We need to update the cracker column and the cracker index Efficiently Maintain the self-organization properties Two issues: When How

105 When to propagate updates in cracking Follow the workload to maintain self-organization Updates become part of query processing When an update arrives, it is not applied For each cracker column there is a pending insertions column and a pending deletions column Pending updates are applied only when a query needs the specific values

106 Updates aware select We extended the cracker select operator to apply the needed updates before cracking The select operator: 1. Search the pending insertions column 2. Search the pending deletions column 3. If Steps 1 or 2 find tuples run an update algorithm 4. Search the cracker index 5. Physically reorganize the cracker column 6. Update the cracker index 7. Return a slice of the cracker column

107 Merging 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >12 Start position: 1 values: >1 Insert a new tuple with value 9 The new tuple belongs to the blue piece 9

108 Merging 7 2 10 29 25 31 57 42 53 Start position: 8 values: >35 Start position: 5 values: >12 Start position: 1 values: >1 Insert a new tuple with value 9 The new tuple belongs to the blue piece 9 Pieces in the cracker column are ordered Tuples inside a piece are not ordered Shifting is not a viable solution

109 Merging by Hopping 7 2 10 29 25 31 42 53 Start position: 8 values: >35 Start position: 4 values: >12 Start position: 1 values: >1 57 9 Insert a new tuple with value 9 We need to make enough room to fit the new tuples

110 Merge Gradually A query merges only the qualifying values, i.e., only the values that it needs for a correct and complete result Average cost increases significantly We avoid the large peaks but... Merge CompletelyMerge Gradually

111 The Ripple Touch only the pieces that are relevant for the current query

112 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 Touch only the pieces that are relevant for the current query

113 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 Select 7<= A< 15 Touch only the pieces that are relevant for the current query

114 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 Select 7<= A< 15 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query

115 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Select 7<= A< 15

116 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Select 7<= A< 15

117 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Select 7<= A< 15

118 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Avoid shifting down non interesting pieces Select 7<= A< 15

119 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Avoid shifting down non interesting pieces Select 7<= A< 15

120 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Immediately make room for the new tuples Avoid shifting down non interesting pieces Select 7<= A< 15

121 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Immediately make room for the new tuples Avoid shifting down non interesting pieces Select 7<= A< 15

122 The Ripple 7 2 10 29 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions Touch only the pieces that are relevant for the current query Immediately make room for the new tuples Avoid shifting down non interesting pieces Select 7<= A< 15

123 The Ripple 7 2 10 25 31 57 42 53 Start position: 7 values: >35 Start position: 4 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions 29 Touch only the pieces that are relevant for the current query Immediately make room for the new tuples Avoid shifting down non interesting pieces Select 7<= A< 15

124 The Ripple 7 2 10 25 31 57 42 53 Start position: 7 values: >35 Start position: 5 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions 29 Touch only the pieces that are relevant for the current query Immediately make room for the new tuples Avoid shifting down non interesting pieces Select 7<= A< 15

125 The Ripple 7 2 10 25 31 57 42 53 Start position: 7 values: >35 Start position: 5 values: >22 Start position: 1 values: >1 5 9 16 35 Pending insertions 29 Touch only the pieces that are relevant for the current query Immediately make room for the new tuples Avoid shifting down non interesting pieces Select 7<= A< 15

126 The Ripple Maintain high performance through the whole query sequence in a self-organizing way

127 The Ripple Maintain high performance through the whole query sequence in a self-organizing way Merge GraduallyMerge Completely Merge Ripple


Download ppt "Distributed Query Processing. Agenda Recap of query optimization Transformation rules for P&D systems Memoization Queries in heterogeneous systems Query."

Similar presentations


Ads by Google