1. CPS216: Advanced Database Systems Notes 03:Query Processing (Overview, contd.)
Shivnath Babu

2. Announcements First homework will be posted this week
I will send an announcement when it is posted
We will start discussing past projects and project ideas from next class
Your first goal is to have a concretely defined project by the last week of September (the earlier the better!)
Try to form groups of two
October and November should be used to work on the projects
Final “demo’s” will be in the first week of December

3. Physical plan generation
Overview of
Query
Processing
SQL query
parse
parse tree
Query rewriting
Query
Optimization
statistics
logical query plan
Physical plan generation
physical query plan
execute
Query
Execution
result

4. Physical plan generation
SQL query
Initial logical plan
parse
Rewrite rules
parse tree
Query rewriting
Logical plan
statistics
logical query plan
Physical plan generation
“Best” logical plan
physical query plan
execute
result

5. B,D B,D R.C = S.C R.A = “c” Λ R.C = S.C R.A = “c” Query RewritingX X R S R S
We will revisit it towards the end of this lecture

6. Physical plan generation
SQL query
parse
parse tree
Query rewriting
statistics
Best logical query plan
Physical plan generation
Best physical query plan
execute
result

7. Physical Plan Generation
B,D
Project
Natural join
Hash join
R.A = “c” S
Index scan
Table scan R R S
Best logical plan

8. Physical plan generation
SQL query
parse
parse tree
Enumerate possible
physical plans
Query rewriting
statistics
Best logical query plan
Find the cost of
each plan
Physical plan generation
Best physical query plan
Pick plan with
minimum cost
execute
result

9. Physical Plan Generation
Logical Query Plan
P1 P2 …. Pn
C1 C2 …. Cn
Pick minimum cost one
Physical
plans
Costs

10. Plans for Query Execution
Roadmap
Path of a SQL query
Operator trees
Physical Vs Logical plans
Plumbing: Materialization Vs pipelining

11. Modern DBMS Architecture
Applications
SQL
DBMS
Parser
Query Optimizer
Query Executor
Storage Manager
Logical query plan
Physical query plan
Access method API calls
Storage system API calls
File system API calls OS
Disk(s)

12. Logical Plans Vs. Physical Plans
B,D
Project
Natural join
Hash join
R.A = “c” S
Index scan
Table scan R R S
Best logical plan

13. Operator Plumbing B,D R.A = “c” S R
Materialization: output of one operator written to disk, next operator reads from the disk
Pipelining: output of one operator directly fed to next operator

14. Materialization
B,D
Materialized here
R.A = “c” S R

15. Iterators: Pipelining
B,D
 Each operator supports:
Open()
GetNext()
Close()
R.A = “c” S R

16. Iterator for Table Scan (R)
Open() {
/** initialize variables */
b = first block of R;
t = first tuple in block b; }
GetNext() {
IF (t is past last tuple in block b) {
set b to next block;
IF (there is no next block)
/** no more tuples */
RETURN EOT;
ELSE t = first tuple in b; }
/** return current tuple */
oldt = t;
set t to next tuple in block b;
RETURN oldt;
Close() {
/** nothing to be done */ }

17. Iterator for Select R.A = “c” GetNext() { LOOP: t = Child.GetNext();
IF (t == EOT) {
/** no more tuples */
RETURN EOT; }
ELSE IF (t.A == “c”)
RETURN t;
ENDLOOP:
Open() {
/** initialize child */
Child.Open(); }
Close() {
/** inform child */
Child.Close(); }

18. Iterator for Sort R.A GetNext() { IF (more tuples)
RETURN next tuple in order;
ELSE RETURN EOT; }
Open() {
/** Bulk of the work is here */
Child.Open();
Read all tuples from Child
and sort them }
Close() {
/** inform child */
Child.Close(); }

19. Iterator for Tuple Nested Loop Join
Lexp
Rexp
TNLJ (conceptually)
for each r  Lexp do
for each s  Rexp do
if Lexp.C = Rexp.C, output r,s

20. Example 1: Left-Deep Plan
TNLJ
TableScan
TNLJ
R3(C,D)
TableScan
TableScan
R1(A,B)
R2(B,C)
Question: What is the sequence of getNext() calls?

21. Example 2: Right-Deep Plan
TNLJ
R3(C,D)
TableScan
R1(A,B)
TableScan
R2(B,C)
TNLJ
Question: What is the sequence of getNext() calls?

22. Example
Worked on blackboard

23. Cost Measure for a Physical Plan
There are many cost measures
Time to completion
Number of I/Os (we will see a lot of this)
Number of getNext() calls
Tradeoff: Simplicity of estimation Vs. Accurate estimation of performance as seen by user

24. Physical plan generation
Query Processing - In class order
parse
Query rewriting
Physical plan generation
execute
result
SQL query
parse tree
logical query plan
statistics
physical query plan
2; 16.1
First: A quick look at this
3; 16.2,16.3
4; 16.4—16.7
1; 13, 15

25. Textbook outline Chapter 15 15.1 Physical operators
- Scan, Sort (Ch. 11.4), Indexes (Ch. 13)
Implementing operators +
estimating their cost
15.8 Buffer Management
15.9 Parallel Processing

26. Textbook outline (contd.)
Chapter 16
16.1 Parsing
16.2 Algebraic laws
16.3 Parse tree  logical query plan
16.4 Estimating result sizes
Cost based optimization

27. Background Material
Chapter 5 Relational Algebra
Chapter 6 SQL

28. Why do we need Query Rewriting?
Pruning the HUGE space of physical plans
Eliminating redundant conditions/operators
Rules that will improve performance with very high probability
Preprocessing
Getting queries into a form that we know how to handle best
Reduces optimization time drastically without noticeably affecting quality

29. Some Query Rewrite Rules
Transform one logical plan into another
Do not use statistics
Equivalences in relational algebra
Push-down predicates
Do projects early
Avoid cross-products if possible

30. Equivalences in Relational Algebra
R S = S R Commutativity
(R S) T = R (S T) Associativity
Also holds for: Cross Products, Union, Intersection
R x S = S x R
(R x S) x T = R x (S x T)
R U S = S U R
R U (S U T) = (R U S) U T

31. B,D B,D R.C = S.C R.A = “c” Λ R.C = S.C R.A = “c”
Apply Rewrite Rule (1)
B,D
B,D
R.C = S.C
R.A = “c” Λ R.C = S.C
R.A = “c” X X R S R S
B,D [ sR.C=S.C [R.A=“c”(R X S)]]

32. pxy (R) = px [py (R)] Rules: Project Let: X = set of attributes
Y = set of attributes
XY = X U Y
pxy (R) =
px [py (R)]

33. sp (R S) = sq (R S) = Rules: s + combined
Let p = predicate with only R attribs
q = predicate with only S attribs
m = predicate with only R,S attribs
sp (R S) =
sq (R S) =
[sp (R)] S
R [sq (S)]

34. B,D B,D R.C = S.C R.C = S.C R.A = “c” R.A = “c”
Apply Rewrite Rule (2)
B,D
B,D
R.C = S.C
R.C = S.C
R.A = “c” X
R.A = “c” S X R S R
B,D [ sR.C=S.C [R.A=“c”(R)] X S]

35. B,D B,D R.C = S.C R.A = “c” R.A = “c” Apply Rewrite Rule (3) X S
Natural join
R.A = “c” X S
R.A = “c” S R R
B,D [[R.A=“c”(R)] S]

36. Rules: s + combined (continued)
spq (R S) = [sp (R)] [sq (S)]
spqm (R S) =
sm [(sp R) (sq S)]
spvq (R S) =
[(sp R) S] U [R (sq S)]

37. Which are “good” transformations?
sp1p2 (R)  sp1 [sp2 (R)]
sp (R S)  [sp (R)] S
R S  S R
px [sp (R)]  px {sp [pxz (R)]}

38. Conventional wisdom: do projects early
Example: R(A,B,C,D,E) P: (A=3)  (B=“cat”)
pE {sp (R)} vs. pE {sp{pABE(R)}}

39. But: What if we have A, B indexes?
B = “cat” A=3
Intersect pointers to get
pointers to matching tuples

40. Bottom line: No transformation is always good Some are usually good:
Push selections down
Avoid cross-products if possible
Subqueries  Joins

41. Avoid Cross Products (if possible)
Select B,D
From R,S,T,U
Where R.A = S.B  R.C=T.C  R.D = U.D
Which join trees avoid cross-products?
If you can't avoid cross products, perform them as late as possible

42. More Query Rewrite Rules
Transform one logical plan into another
Do not use statistics
Equivalences in relational algebra
Push-down predicates
Do projects early
Avoid cross-products if possible
Use left-deep trees
Subqueries  Joins (we will study this rewrite rule after we do physical plan selection)
Use of constraints, e.g., uniqueness