CS 347Notes 041 CS 347: Distributed Databases and Transaction Processing Notes04: Query Optimization Hector Garcia-Molina

CS 347Notes 042 Query optimization Cost estimation Strategies for exploring plans min Q

CS 347Notes 043 Cost estimation  As in centralized system: estimate result sizes

CS 347Notes 044  But: # IOs may not be best metric e.g., Transmission time may dominate workworkanswerat site T1T2 >>> TIME > or $

CS 347Notes 045 Another reason why plain IOs not enough:  Parallelism Plan APlan B site 150 IOs 100 IOs site 270 IOs site 350 IOs

CS 347Notes 046 Cost metrics –IOs, Bytes transmitted, $, … –Can add together Response time metric –cannot add –need scheduling and dependency info –skew important Task 1 Task 2 Task 3

CS 347Notes 047  Take into account: (in parallel/distributed system) Start up costs (for parallel operation) Data distribution costs/time Contention –memory, disk, network,… Assembling result

CS 347Notes 048 Example: Response time Site 1 Site 2 Site 3 Site 4 StartupDistri- Searching Final bution +send results proc.

CS 347Notes 049 Searching strategies (1)Exhaustive (with pruning) (2)Hill climbing (greedy) (3)Query separation

CS 347Notes 0410 (1) Exhaustive - consider “all” query plans with a set of techniques - prune some plans - heuristics

CS 347Notes 0411 RST R S R  T S R S T T S T  R (S R) T (T S) R ship S semi ship T semi to R join to Sjoin 1Prune because cross-product not necessary 2Prune because larger relation first Example: join |R|>|S|>|T| RST AB 2121

CS 347Notes 0412  In generating plans, keep goal in mind: e.g.: Goal is parallelism in system with fast net, consider partitioning relation(s) first e.g.: Goal is reduction of net traffic, consider semi-joins

CS 347Notes 0413 (2) Hill climbing Better plans Worse plans x Initial plan 1

CS 347Notes 0414 (2) Hill climbing Better plans Worse plans x Initial plan 1 2

CS 347Notes 0415 Example R S T V RelSiteSizetuple size = 1 R 1 10 S 2 20 T 3 30 V 4 40 RSTV ABC Goal: minimize data transmission

CS 347Notes 0416 Initial plan: send relations to one site What site do we send all relations to? To site 1: cost= =90 To site 2: cost= =80 To site 3: cost= =70 To site 4: cost= =60 

CS 347Notes 0417 P 0 : R (1  4) S (2  4) T (3  4) Compute R S T V at site 4

CS 347Notes 0418 Local search Consider sending each relation to neighbor: e.g.: R S R R S

CS 347Notes 0419 Assume: SizeR S = 20 S T = 5 T V = 1 Option (a) cost = 30cost = 30 No savings R S R S R

CS 347Notes 0420 Option (b) cost = 30cost = 40 Worse off! S R S S R

CS 347Notes 0421 Option (c) cost = 50cost = 35 A Win! T S T T S

CS 347Notes 0422 Option (d) cost = 50cost = 25 A Bigger Win! S T S T S

CS 347Notes 0423 P 1 : P 1 a: S (2  3)  = S T P 1 b: R (1  4)  (3  4) compute answer at site 4

CS 347Notes 0424 Repeat local search - Treat  = S T as relation R   R  R R  vs

CS 347Notes 0425 Hill climbing may miss best plan! Example: best plan could be: P B : T (3  4)  =T V  (4  2)  ’=  S  ’ (2  1)  ”=  ’ R  ” (1  4) Compute answer  ’’ ’’  V T R S T [optional]

CS 347Notes 0426 Hill climbing may miss best plan! Example: best plan could be: P B : T (3  4)  =T V  (4  2)  ’=  S  ’ (2  1)  ”=  ’ R  ” (1  4) Compute answer  ’’ ’’  V T R S T  30 1 1 1 1 1 1 33 = total Costs could be low because β is very selective [optional]

CS 347Notes 0427 (3) Query separation - separate query into 2 or more steps -optimize each step independently

CS 347Notes 0428 Example: simple queries e.g.:  c 1  c 2  c 3 R S 1. Compute R’ =  A [  c 2 R] S’ =  A [  c 3 S] 2. Compute J = R’ S’ A

CS 347Notes 0429  c 1  c 2  c 3 R S A 1. Compute R’ =  A [  c 2 R] S’ =  A [  c 3 S] 2. Compute J = R’ S’ 3. Compute Ans =  c 1 { [J  c 2 R] [J  c 3 S] }

CS 347Notes 0430 In other words: (a) Compute A values in answer (steps 1,2) (b) Get tuples from sites with matching A values and compute answer (step 3)

CS 347Notes 0431 Simple query - Relations have a single attribute - Output has a single attribute e.g., J  R’ S’

CS 347Notes 0432 Idea Decompose query into –Local processing –Simple query (or queries) –Final processing Optimize simple query

CS 347Notes 0433 Philosophy Hard part is distributed join Do this part with only keys; get rest of data later Simpler to optimize simple queries

CS 347Notes 0434 Summary: Query Optimization Cost estimation Strategies –Exhaustive –Hill climbing –Separation

CS 347Notes 0435 Words of wisdom “Optimization is like chess playing” i.e., May have to make sacrifices (move data, partition relations, build indexes) for later gains!