1. The Query Compiler. 16. 1 Parsing and Preprocessing. Meghna Jain(205)
The Query Compiler Parsing and Preprocessing Meghna Jain(205) Dr. T. Y. Lin

2. Presentation Outline 16.1 Parsing and Preprocessing
Syntax Analysis and Parse Tree
A Grammar for Simple Subset of SQL
The Preprocessor
Processing Queries Involving Views

3. Query compilation is divided into three steps
1. Parsing: Parse SQL query into parser tree.
2. Logical query plan: Transforms parse tree into expression tree of relational algebra.
3.Physical query plan: Transforms logical query plan into physical query plan.
. Operation performed
. Order of operation
. Algorithm used
. The way in which stored data is obtained and passed from one
operation to another.

4. Form a query to a logical query plan
Parser
Preprocessor
Logical Query plan generator
Query rewrite
Preferred logical query plan
Form a query to a logical query plan

5. Syntax Analysis and Parse Tree
Parser takes the sql query and convert it to parse
tree. Nodes of parse tree:
1. Atoms: known as Lexical elements such as key words, constants, parentheses, operators, and
other schema elements.
2. Syntactic categories: Subparts that plays a
similar role in a query as <Query> , <Condition>

6. Grammar for Simple Subset of SQL
<Query> ::= <SFW>
<Query> ::= (<Query>)‏
<SFW> ::= SELECT <SelList> FROM <FromList> WHERE <Condition>
<SelList> ::= <Attribute>,<SelList>
<SelList> ::= <Attribute>
<FromList> ::= <Relation>, <FromList>
<FromList> ::= <Relation>
<Condition> ::= <Condition> AND <Condition>
<Condition> ::= <Tuple> IN <Query>
<Condition> ::= <Attribute> = <Attribute>
<Condition> ::= <Attribute> LIKE <Pattern>
<Tuple> ::= <Attribute>
Atoms(constants), <syntactic categories>(variable),
::= (can be expressed/defined as)‏

7. Query and Parse T ree StarsIn(title,year,starName)
MovieStar(name,address,gender,birthdate)‏
Query:
Give titles of movies that have at least one star born in 1960
SELECT title FROM StarsIn WHERE starName IN (
SELECT name FROM MovieStar WHERE
birthdate LIKE '%1960%' );

9. Another query equivalent
SELECT title
FROM StarsIn, MovieStar
WHERE starName = name AND
birthdate LIKE '%1960%' ;

10. Parse Tree <Query> <SFW>
SELECT <SelList> FROM <FromList> WHERE <Condition>
<Attribute> <RelName> , <FromList> AND
title StarsIn <RelName>
MovieStar
<Query>
<Condition> <Condition>
<Attribute> = <Attribute> <Attribute> LIKE <Pattern>
starName name birthdate ‘%1960’

11. The Preprocessor Functions of Preprocessor
. If a relation used in the query is virtual view then each use of this relation in the form-list must replace by parser tree that describe the view.
. It is also responsible for semantic checking
1. Checks relation uses : Every relation mentioned in FROM-
clause must be a relation or a view in current schema.
2. Check and resolve attribute uses: Every attribute mentioned
in SELECT or WHERE clause must be an attribute of same relation in the current scope.
3. Check types: All attributes must be of a type appropriate to
their uses.

12. StarsIn(title,year,starName)
MovieStar(name,address,gender,birthdate)‏
Query:
Give titles of movies that have at least one star born in 1960
SELECT title FROM StarsIn WHERE starName IN (
SELECT name FROM MovieStar WHERE
birthdate LIKE '%1960%' );

13. Preprocessing Queries Involving Views
When an operand in a query is a virtual view, the preprocessor needs to replace the operand by a piece of parse tree that represents how the view is constructed from base table.
Base Table: Movies( title, year, length, genre, studioname, producerC#)‏
View definition : CREATE VIEW ParamountMovies AS
SELECT title, year FROM movies
WHERE studioName = 'Paramount';
Example based on view:
SELECT title FROM ParamountMovies WHERE year = 1979;

14. Query Compiler [Figure 16.16 & Figure16.18]
CS 257 [section 1]
Aakanksha Pendse.
Roll number : 127

15. To verify the claim given in the figure 16.18 using the figure 16.16
Problem Statement
To verify the claim given in the figure using the figure 16.16

16. What is figure 16.16?
The tree in this diagram represents the following query.
“find movies with movie-stars born in 1960”
Database tables with its attributes are:
StarsIn(movieTitle, movieYear, starName)
MovieStar(name, address, gender, birthdate)
SQL Query:
SELECT movieTitle
FROM StarsIn
WHERE starName IN (
SELECT name
FROM MovieStar
WHERE birthdate LIKE ‘%1960’ );

17. This diagram (fig.16.14) is derived from figure 16.14
This is done by applying the Rule which handles the two-argument selection with a condition involving IN
In this query the sub-query is uncorrelated.

18. Figure 16.14

19. Figure 16.16 : applying the rule for IN condition

20. Rule

21. What is figure 16.18?
The tree in this diagram represents the following query.
“Find the movies where the average age of the stars was at most 40 when the movie was made.”
SQL Query :
SELECT DISTINCT m1.movieTitle, m1.movieYear FROM StarsIn m1 WHERE m1.movieYear – 40 <= (SELECT AVG(birthdate) FROM StarsIn m2, MovieStar s WHERE m2.starName = s.name AND m1.movieTitle = m2.movieTitle AND m1.movieYear = m2.movieYear);
This is a co-related sub-query.

22. Rules for translating a co-related sub-query to relational algebra.
Correlated sub queries contains unknown values defined outside themselves.
Because of this reason, co-related sub-queries cannot be translated in isolation.
These types of sub queries need to be translated so that they produce a relation in which certain extra attributes appear. These attributes must later be compared with the externally defined attributes.
Conditions that relate attributes from the sub query to attributes outside are then applied to this relation. The extra attributes which are not necessary, can be projected out.
In this strategy, care should be taken of not forming duplicate tuples at the end.

23. Steps of formation of tree in figure 16.18
The tree in figure is formed by parsing of the query and partial translation to relational algebra.
The WHERE-clause of the sub query is split into two. It is used to covert the product of relations to an equijoin
The aliases m1, m2 and s are made the nodes of the tree.

24. Figure 16.18 : partially transformed parse tree

25. 16.2 ALGEBRAIC LAWS FOR IMPROVING QUERY PLANS
Ramya Karri
ID: 206

26. Optimizing the Logical Query Plan
The translation rules converting a parse tree to a logical query tree do not always produce the best logical query tree.
It is often possible to optimize the logical query tree by applying relational algebra laws to convert the original tree into a more efficient logical query tree.
Optimizing a logical query tree using relational algebra laws is called heuristic optimization

27. Relational Algebra Laws
These laws often involve the properties of:
commutativity - operator can be applied to operands independent of order.
E.g. A + B = B + A - The “+” operator is commutative.
associativity - operator is independent of operand grouping.
E.g. A + (B + C) = (A + B) + C - The “+” operator is associative.

28. Associative and Commutative Operators
The relational algebra operators of cross-product (×), join (⋈), union, and intersection are all associative and commutative.
Commutative
R X S = S X R
R ⋈ S = S ⋈ R
R  S = S  R
R ∩ S = S ∩ R
Associative
(R X S) X T = S X (R X T)
(R ⋈ S) ⋈ T= S ⋈ (R ⋈ T)
(R  S)  T = S  (R  T)
(R ∩ S) ∩ T = S ∩ (R ∩ T)

29. Laws Involving Selection
Complex selections involving AND or OR can be broken into two or more selections: (splitting laws)
σC1 AND C2 (R) = σC1( σC2 (R))
σC1 OR C2 (R) = ( σC1 (R) ) S ( σC2 (R) )
Example
R={a,a,b,b,b,c}
p1 satisfied by a,b, p2 satisfied by b,c
σp1vp2 (R) = {a,a,b,b,b,c}
σp1(R) = {a,a,b,b,b}
σp2(R) = {b,b,b,c}
σp1 (R) U σp2 (R) = {a,a,b,b,b,c}

30. Laws Involving Selection (Contd..)
Selection is pushed through both arguments for union:
σC(R  S) = σC(R)  σC(S)
Selection is pushed to the first argument and optionally the second for difference:
σC(R - S) = σC(R) - S
σC(R - S) = σC(R) - σC(S)

31. Laws Involving Selection (Contd..)
All other operators require selection to be pushed to only one of the arguments.
For joins, may not be able to push selection to both if argument does not have attributes selection requires.
σC(R × S) = σC(R) × S
σC(R ∩ S) = σC(R) ∩ S
σC(R ⋈ S) = σC(R) ⋈ S
σC(R ⋈D S) = σC(R) ⋈D S

32. Laws Involving Selection (Contd..)
Example
Consider relations R(a,b) and S(b,c) and the expression
σ (a=1 OR a=3) AND b<c (R ⋈S)
σ a=1 OR a=3(σ b<c (R ⋈S))
σ a=1 OR a=3(R ⋈ σ b<c (S))
σ a=1 OR a=3(R) ⋈ σ b<c (S)

33. Laws Involving Projection
Like selections, it is also possible to push projections down the logical query tree. However, the performance gained is less than selections because projections just reduce the number of attributes instead of reducing the number of tuples.

34. Laws Involving Projection
Laws for pushing projections with joins:
πL(R × S) = πL(πM(R) × πN(S))
πL(R ⋈ S) = πL((πM(R) ⋈ πN(S))
πL(R ⋈D S) = πL((πM(R) ⋈D πN(S))

35. Laws Involving Projection
Laws for pushing projections with set operations.
Projection can be performed entirely before union.
πL(R UB S) = πL(R) UB πL(S)
Projection can be pushed below selection as long as we also keep all attributes needed for the selection (M = L  attr(C)).
πL ( σC (R)) = πL( σC (πM(R)))

36. Laws Involving Join
We have previously seen these important rules about joins:
Joins are commutative and associative.
Selection can be distributed into joins.
Projection can be distributed into joins.

37. Laws Involving Duplicate Elimination
The duplicate elimination operator (δ) can be pushed through many operators.
R has two copies of tuples t, S has one copy of t,
δ (RUS)=one copy of t
δ (R) U δ (S)=two copies of t

38. Laws Involving Duplicate Elimination
Laws for pushing duplicate elimination operator (δ):
δ(R × S) = δ(R) × δ(S)
δ(R S) = δ(R) δ(S)
δ(R D S) = δ(R) D δ(S)
δ( σC(R) = σC(δ(R))

39. Laws Involving Duplicate Elimination
The duplicate elimination operator (δ) can also be pushed through bag intersection, but not across union, difference, or projection in general.
δ(R ∩ S) = δ(R) ∩ δ(S)

40. Laws Involving Grouping
The grouping operator (γ) laws depend on the aggregate operators used.
There is one general rule, however, that grouping subsumes duplicate elimination:
δ(γL(R)) = γL(R)
The reason is that some aggregate functions are unaffected by duplicates (MIN and MAX) while other functions are (SUM, COUNT, and AVG).

41. Query Compiler By:Payal Gupta Roll No:106(225)
Professor :Tsau Young Lin

42. Pushing Selections
It is, replacing the left side of one of the rules by its right side.
In pushing selections we first a selection as far up the tree as it would go, and then push the selections down all possible branches.

43. Let’s take an example:
S t a r s I n ( t i t l e , year, starName)
Movie(title, year, length, incolor, studioName, producerC#)
Define view MoviesOf 1996 by:
CREATE VIEW MoviesOfl996 AS
SELECT *
FROM Movie
,WHERE year = 1996;

44. "which stars worked for which studios in 1996
"which stars worked for which studios in 1996?“ can be given by a SQL Query:
SELECT starName, studioName
FROM MoviesOfl996 NATURAL JOIN StarsIn;

45. ΠstarName,studioName O StarsIn Year=1996 Movie
Logical query plan constructed from definition of a query and view

46. Improving the query plan by moving selections up and down the tree
ΠstarName,studioName O O
Year=1996
Year=1996
StarsIn
Movie

47. Laws Involving Projection
"pushing" projections really involves introducing a new projection somewhere below an existing projection.
projection keeps the number of tuples the same and only reduces the length of tuples.
To describe the transformations of extended projection Consider a term E + x on the list for a projection, where E is an attribute or an expression involving attributes and constants and x is an output attribute.

48. Example
Let R(a, b, c) and S(c, d, e) be two relations. Consider the expression x,+,,,, b+y(R w S). The input attributes of the projection are a,b, and e, and c is the only join attribute. We may apply the law for pushing projections below joins to get the equivalent expression:
Πa+e->x,b->y(Πa,b,c(R) Πc,e(S))
Eliminating this projection and getting a third equivalent expression:Πa+e->x, b->y( R Πc,e(S))

49. In addition, we can perform a projection entirely before a bag union
In addition, we can perform a projection entirely before a bag union. That is:
ΠL(R UB S)= ΠL(R) )UB ΠL(S)

50. Laws About Joins and Products
laws that follow directly from the definition of the join:
R c S = c( R * S)
R S = ΠL( c ( R * S) ) , where C is the condition that equates each pair of attributes from R and S with the same name. and L is a list that includes one attribute from each equated pair and all the other attributes of R and S.
We identify a product followed by a selection as a join of some kind. O O

51. Laws Involving Duplicate Elimination
The operator δ which eliminates duplicates from a bag can be pushed through many but not all operators.
In general, moving a δ down the tree reduces the size of intermediate relations and may therefore beneficial.
Moreover, sometimes we can move δ to a position where it can be eliminated altogether,because it is applied to a relation that is known not to possess duplicates.

52. δ (R)=R if R has no duplicates
δ (R)=R if R has no duplicates. Important cases of such a relation R include:
a) A stored relation with a declared primary key, and
b) A relation that is the result of a γ operation, since grouping creates a relation with no duplicates.

53. Several laws that "push" δ through other operators are:
δ (R*S) =δ(R) * δ(S)
δ (R S)=δ(R) δ(S)
δ (R c S)=δ(R) c δ(S)
δ ( c (R))= c (δ(R))
We can also move the δ to either or both of the arguments of an intersection:
δ (R ∩B S) = δ(R) ∩B S = R ∩B δ (S) = δ(R) ∩B δ (S) O O

54. Laws Involving Grouping and Aggregation
When we consider the operator γ, we find that the applicability of many transformations depends on the details of the aggregate operators used. Thus we cannot state laws in the generality that we used for the other operators. One exception is that a γ absorbs a δ . Precisely:
δ(γL(R))=γL(R)

55. let us call an operator γ duplicate-impervious if the only aggregations in L are MIN and/or MAX then:
γ L(R) = γ L (δ(R)) provided γL is duplicate- impervious.

56. Example Suppose we have the relations
MovieStar(name , addr , gender, birthdate)
StarsIn(movieTitle, movieyear, starname)
and we want to know for each year the birthdate of the youngest star to appear in a movie that year. We can express this query as:
SELECT movieyear, MAX(birth date)
FROM MovieStar, StarsIn
WHERE name = starName
GROUP BY movieyear;

57. γ movieYear, MAX ( birthdate ) name = starName
MovieStar StarsIn
Initial logical query plan for the query O

58. Some transformations that we can apply to Fig are
1. Combine the selection and product into an equijoin.
2.Generate a δ below the γ , since the γ is duplicate impervious.
3. Generate a Π between the γ and the introduced δ to project onto movie-Year and birthdate, the only attributes relevant to the γ

59. γ movieYear, MAX ( birthdate )
Π movieYear, birthdate δ
name = starName
MovieStar StarsIn
Another query plan for the query

60. γ movieYear, MAX ( birthdate )
Π movieYear, birthdate
name = starName
δ δ
Π birthdate,name Π movieYear,starname
MovieStar StarsIn
third query plan for Example

61. CS 255: Database System Principles slides: From Parse Trees to Logical Query Plans
By:- Arunesh Joshi
Id:

62. Agenda Conversion to Relational Algebra.
Removing Sub queries From Conditions.
Improving the Logical Query Plan.
Grouping Associative/Commutative Operators.

63. Parsing
Goal is to convert a text string containing a query into a parse tree data structure:
leaves form the text string (broken into lexical elements)
internal nodes are syntactic categories
Uses standard algorithmic techniques from compilers
given a grammar for the language (e.g., SQL), process the string and build the tree

64. Example: SQL query Assume we have a simplified grammar for SQL.
SELECT title
FROM StarsIn
WHERE starName IN (
SELECT name
FROM MovieStar
WHERE birthdate LIKE ‘%1960’ );
(Find the movies with stars born in 1960)
Assume we have a simplified grammar for SQL.

65. SELECT <SelList> FROM <FromList> WHERE <Condition>
Example: Parse Tree
<Query>
<SFW>
SELECT <SelList> FROM <FromList> WHERE <Condition>
<Attribute> <RelName> <Tuple> IN <Query>
title StarsIn <Attribute> ( <Query> )
starName <SFW>
SELECT <SelList> FROM <FromList> WHERE <Condition>
<Attribute> <RelName> <Attribute> LIKE <Pattern>
name MovieStar birthDate ‘%1960’

66. The Preprocessor
It replaces each reference to a view with a parse (sub)-tree that describes the view (i.e., a query)
It does semantic checking:
are relations and views mentioned in the schema?
are attributes mentioned in the current scope?
are attribute types correct?

67. Convert Parse Tree to Relational Algebra
The complete algorithm depends on specific grammar, which determines forms of the parse trees
Here is a flavor of the approach

68. Conversion att-list(cond( X (rel-list)))
Suppose there are no subqueries.
SELECT att-list FROM rel-list WHERE cond
is converted into
PROJatt-list(SELECTcond(PRODUCT(rel-list))), or
att-list(cond( X (rel-list)))

69. SELECT <SelList> FROM <FromList> WHERE <Condition>
SELECT movieTitle
FROM StarsIn, MovieStar
WHERE starName = name AND birthdate LIKE '%1960';
<Query>
<SFW>
SELECT <SelList> FROM <FromList> WHERE <Condition>
<Attribute> <RelName> , <FromList> AND <Condition>
movieTitle StarsIn <RelName> <Attribute> LIKE <Pattern>
MovieStar birthdate '%1960'
<Condition>
<Attribute> = <Attribute>
starName name

70. Equivalent Algebraic Expression Tree
movieTitle 
starname = name AND birthdate LIKE '%1960' X
StarsIn MovieStar

71. Handling Subqueries Recall the (equivalent) query:
SELECT title
FROM StarsIn
WHERE starName IN (
SELECT name
FROM MovieStar
WHERE birthdate LIKE ‘%1960’ );
Use an intermediate format called two-argument selection

72. title  Example: Two-Argument Selection StarsIn <condition>
<tuple> IN name
<attribute> birthdate LIKE ‘%1960’
starName MovieStar

73. Converting Two-Argument Selection
To continue the conversion, we need rules for replacing two-argument selection with a relational algebra expression
Different rules depending on the nature of the sub query
Here is shown an example for IN operator and uncorrelated query (sub query computes a relation independent of the tuple being tested)

74. Rules for IN   C R <Condition> X R  t IN S S
C is the condition that equates
attributes in t with corresponding
attributes in S

75. title starName=name  birthdate LIKE ‘%1960’
Example: Logical Query Plan
title
starName=name 
StarsIn name
birthdate LIKE ‘%1960’
MovieStar

76. What if Subquery is Correlated?
Example is when subquery refers to the current tuple of the outer scope that is being tested
More complicated to deal with, since subquery cannot be translated in isolation
Need to incorporate external attributes in the translation
Some details are in textbook

77. Improving the Logical Query Plan
There are numerous algebraic laws concerning relational algebra operations
By applying them to a logical query plan judiciously, we can get an equivalent query plan that can be executed more efficiently
Next we'll survey some of these laws

78. title birthdate LIKE ‘%1960’ Example: Improved Logical Query Plan
starName=name
StarsIn name
birthdate LIKE ‘%1960’
MovieStar

79. Associative and Commutative Operations
product
natural join
set and bag union
set and bag intersection
associative: (A op B) op C = A op (B op C)
commutative: A op B = B op A

80. Laws Involving Selection
Selections usually reduce the size of the relation
Usually good to do selections early, i.e., "push them down the tree"
Also can be helpful to break up a complex selection into parts

81. Selection Splitting  C1 AND C2 (R) =  C1 (  C2 (R))
 C1 OR C2 (R) = ( C1 (R)) Uset ( C2 (R))
if R is a set
 C1 (  C2 (R)) =  C2 (  C1 (R))

82. Selection and Binary Operators
Must push selection to both arguments:
 C (R U S) =  C (R) U  C (S)
Must push to first arg, optional for 2nd:
 C (R - S) =  C (R) - S
 C (R - S) =  C (R) -  C (S)
Push to at least one arg with all attributes mentioned in C:
product, natural join, theta join, intersection
e.g.,  C (R X S) =  C (R) X S, if R has all the atts in C

83. Pushing Selection Up the Tree
Suppose we have relations
StarsIn(title,year,starName)
Movie(title,year,len,inColor,studioName)
and a view
CREATE VIEW MoviesOf1996 AS
SELECT *
FROM Movie
WHERE year = 1996;
and the query
SELECT starName, studioName
FROM MoviesOf1996 NATURAL JOIN StarsIn;

84. The Straightforward Tree
starName,studioName
year= StarsIn
Remember the rule
C(R S) = C(R) S ?
Movie

85. The Improved Logical Query Plan
starName,studioName
year= StarsIn
Movie
starName,studioName
year=1996
Movie
StarsIn
starName,studioName
year= year=1996
Movie
StarsIn
push selection
up tree
push selection
down tree

86. Grouping Assoc/Comm Operators
Groups together adjacent joins, adjacent unions, and adjacent intersections as siblings in the tree
Sets up the logical QP for future optimization when physical QP is constructed: determine best order for doing a sequence of joins (or unions or intersections)
U D E F U D E F U
A B C A B C

87. 16.4 Estimating the Cost of Operations
Project Guide Prepared By
Dr. T. Y. Lin Akshay Shenoy
Computer Science Dept
San Jose State University

88. Introduction Possible Physical Plan
Estimating Sizes of Intermediate Relations
Estimating the Size of a Projection
Estimating the Size of a Selection
Estimating the Size of a Join
Natural Joins With Multiple Join Attributes
Joins of Many Relations
Estimating Sizes of Other Operations

89. Physical Plan For each physical plan select
An order and grouping for associative-and-commutative operations like joins, unions.
An Algorithm for each operator in the logical plan.
eg: whether nested loop join or hash join to be used
Additional operators that are needed for the physical plan but that were not present explicitly in the logical plan. eg: scanning, sorting
The way in which arguments are passed from one operator to the next.

90. Estimating Sizes of Intermediate Relations
Rules for estimating the number of tuples in an intermediate relation:
Give accurate estimates
Are easy to compute
Are logically consistent
Objective of estimation is to select best physical plan with least cost.

91. Estimating the Size of a Projection
The projection is different from the other operators, in that the size of the result is computable. Since a projection produces a result tuple for every argument tuple, the only change in the output size is the change in the lengths of the tuples.

92. Estimating the Size of a Selection(1)
Let , ,where A is an attribute of R and C is a constant. Then we recommend as an estimate:
T(S) =T(R)/V(R,A)
The rule above surely holds if all values of attribute A occur equally often in the database.

93. Estimating the Size of a Selection(2)
If ,then our estimate for
T(s) is: T(S) = T(R)/3
We may use T(S)=T(R)(V(R,a) -1 )/ V(R,a) as an estimate.
When the selection condition C is the And of several equalities and inequalities, we can treat the selection as a cascade of simple selections, each of which checks for one of the conditions.

94. Estimating the Size of a Selection(3)
A less simple, but possibly more accurate estimate of the size of is to assume that C1 and of which satisfy C2, we would estimate the number of tuples in S as
In explanation, is the fraction of tuples that do not satisfy C1, and is the fraction that do not satisfy C2. The product of these numbers is the fraction of R’s tuples that are not in S, and 1 minus this product is the fraction that are in S.

95. Estimating the Size of a Join
two simplifying assumptions:
1. Containment of Value Sets
If R and S are two relations with attribute Y and V(R,Y)<=V(S,Y) then every Y-value of R will be a Y-value of S.
2. Preservation of Value Sets
Join a relation R with another relation S with attribute A in R and not in S then all distinct values of A are preserved and not lost.V(S R,A) = V(R,A)
Under these assumptions, we estimate
T(R S) = T(R)T(S)/max(V(R,Y), V(S, Y))

96. Natural Joins With Multiple Join Attributes
Of the T(R),T(S) pairs of tuples from R and S, the expected number of pairs that match in both y1 and y2 is:
T(R)T(S)/max(V(R,y1), V(S,y1)) max(V(R, y2), V(S, y2))
In general, the following rule can be used to estimate the size of a natural join when there are any number of attributes shared between the two relations.
The estimate of the size of R S is computed by multiplying T(R) by T(S) and dividing by the largest of V(R,y) and V(S,y) for each attribute y that is common to R and S.

97. Joins of Many Relations(1)
rule for estimating the size of any join
Start with the product of the number of tuples in each relation. Then, for each attribute A appearing at least twice, divide by all but the least of V(R,A)’s.
We can estimate the number of values that will remain for attribute A after the join. By the preservation-of-value-sets assumption, it is the least of these V(R,A)’s.

98. Joins of Many Relations(2)
Based on the two assumptions-containment and preservation of value sets:
No matter how we group and order the terms in a natural join of n relations, the estimation of rules, applied to each join individually, yield the same estimate for the size of the result. Moreover, this estimate is the same that we get if we apply the rule for the join of all n relations as a whole.

99. Estimating Sizes for Other Operations
Union: the average of the sum and the larger.
Intersection:
approach1: take the average of the extremes, which is the half the smaller.
approach2: intersection is an extreme case of the natural join, use the formula
T(R S) = T(R)T(S)/max(V(R,Y), V(S, Y))

100. Estimating Sizes for Other Operations
Difference: T(R)-(1/2)*T(S)
Duplicate Elimination: take the smaller of (1/2)*T(R) and the product of all the V(R, )’s.
Grouping and Aggregation: upper-bound the number of groups by a product of V(R,A)’s, here attribute A ranges over only the grouping attributes of L. An estimate is the smaller of (1/2)*T(R) and this product.

101. 16.5 Introduction to Cost-based plan selection
Amith KC
Student Id: 109

102. Whether selecting a logical query plan or constructing a physical query plan from a logical plan, the query optimizer needs to estimate the cost of evaluating certain expressions.
We shall assume that the "cost" of evaluating an expression is approximated well by the number of disk I/O's performed.

103. The number of disk I/O’s, in turn, is influenced by:
1. The particular logical operators chosen to implement the query, a matter decided when we choose the logical query plan.
2. The sizes of intermediate results (whose estimation we discussed in Section 16.4)
3. The physical operators used to implement logical operators. e.g.. The choice of a one-pass or two-pass join, or the choice to sort or not sort a given relation.
4. The ordering of similar operations, especially joins
5. The method of passing arguments from one physical operator to the next.

104. Obtaining Estimates for Size Parameter
The formulas of Section 16.4 were predicated on knowing certain important parameters, especially T(R), the number of tuples in a relation R, and V(R, a), the number of different values in the column of relation R for attribute a.
A modern DBMS generally allows the user or administrator explicitly to request the gathering of statistics, such as T(R) and V(R, a). These statistics are then used in subsequent query optimizations to estimate the cost of operations.
By scanning an entire relation R, it is straightforward to count the number of tuples T(R) and also to discover the number of different values V(R, a) for each attribute a.
The number of blocks in which R can fit, B(R), can be estimated either by counting the actual number of blocks used (if R is clustered), or by dividing T(R) by the number of tuples per block

105. Computation of Statistics
Periodic re-computation of statistics is the norm in most DBMS's, for several reasons.
First, statistics tend not to change radically in a short time.
Second, even somewhat inaccurate statistics are useful as long as they are applied consistently to all the plans.
Third, the alternative of keeping statistics up-to-date can make the statistics themselves into a "hot-spot" in the database; because statistics are read frequently, we prefer not to update them frequently too.

106. The recomputation of statistics might be triggered automatically after some period of time, or after some number of updates.
However, a database administrator noticing, that poor- performing query plans are being selected by the query optimizer on a regular basis, might request the recomputation of statistics in an attempt to rectify the problem.
Computing statistics for an entire relation R can be very expensive, particularly if we compute V(R, a) for each attribute a in the relation.
One common approach is to compute approximate statistics by sampling only a fraction of the data. For example, let us suppose we want to sample a small fraction of the tuples to obtain an estimate for V(R, a).

107. Heuristics for Reducing the Cost of Logical Query Plans
One important use of cost estimates for queries or sub-queries is in the application of heuristic transformations of the query.
We have already observed previously how certain heuristics applied independent of cost estimates can be expected almost certainly to improve the cost of a logical query plan.
However, there are other points in the query optimization process where estimating the cost both before and after a transformation will allow us to apply a transformation where it appears to reduce cost and avoid the transformation otherwise.
In particular, when the preferred logical query plan is being generated, we may consider a number of optional transformations and the costs before and after.

108. Because we are estimating the cost of a logical query plan, so we have not yet made decisions about the physical operators that will be used to implement the operators of relational algebra, our cost estimate cannot be based on disk I/Os.
Rather, we estimate the sizes of all intermediate results using the techniques of Section 16.1, and their sum is our heuristic estimate for the cost of the entire logical plan.
For example,

109. Consider the initial logical query plan of as shown below,

110. The statistics for the relations R and S be as follows
To generate a final logical query plan from, we shall insist that the selection be pushed down as far as possible. However, we are not sure whether it makes sense to push the δ below the join or not. Thus, we generate from the two query plans shown in next slide. They differ in whether we have chosen to eliminate duplicates before or after the join.

111. (a)
(b)

112. We know how to estimate the size of the result of the selections, we divide T(R) by V(R, a) = 50.
We also know how to estimate the size of the joins; we multiply the sizes of the arguments and divide by max(V(R, b), V(S, b)), which is 200.

113. Approaches to Enumerating Physical Plans
Let us consider the use of cost estimates in the conversion of a logical query plan to a physical query plan.
The baseline approach, called exhaustive, is to consider all combinations of choices (for each of issues like order of joins, physical implementation of operators, and so on).
Each possible physical plan is assigned an estimated cost, and the one with the smallest cost is selected.

114. There are two broad approaches to exploring the space of possible physical plans:
Top-down: Here, we work down the tree of the logical query plan from the root.
Bottom-up: For each sub-expression of the logical-query-plan tree, we compute the costs of all possible ways to compute that sub-expression. The possibilities and costs for a sub-expression E are computed by considering the options for the sub-expressions for E, and combining them in all possible ways with implementations for the root operator of E.

115. Branch-and-Bound Plan Enumeration
This approach, often used in practice, begins by using heuristics to find a good physical plan for the entire logical query plan. Let the cost of this plan be C. Then as we consider other plans for sub-queries, we can eliminate any plan for a sub-query that has a cost greater than C, since that plan for the sub-query could not possibly participate in a plan for the complete query that is better than what we already know.
Likewise, if we construct a plan for the complete query that has cost less than C, we replace C by the cost of this better plan in subsequent exploration of the space of physical query plans.

116. Hill Climbing
This approach, in which we really search for a “valley” in the space of physical plans and their costs; starts with a heuristically selected physical plan.
We can then make small changes to the plan, e.g., replacing one method for an operator by another, or reordering joins by using the associative and/or commutative laws, to find "nearby" plans that have lower cost.
When we find a plan such that no small modification yields a plan of lower cost, we make that plan our chosen physical query plan.

117. Dynamic Programming
In this variation of the general bottom-UP strategy, we keep for each sub-expression only the plan of least cost.
As we work UP the tree, we consider possible implementations of each node, assuming the best plan for each sub-expression is also used.

118. Selinger-Style Optimization
This approach improves upon the dynamic-programming approach by keeping for each sub-expression not only the plan of least cost, but certain other plans that have higher cost, yet produce a result that is sorted in an order that may be useful higher up in the expression tree. Examples of such interesting orders are when the result of the sub-expression is sorted on one of:
The attribute(s) specified in a sort (r) operator at the root
The grouping attribute(s) of a later group-by (γ) operator.
The join attribute(s) of a later join.

119. Choosing an Order for Joins
Chapter 16.6 by:
Chiu Luk
ID: 210

120. Introduction
This section focuses on critical problem in cost-based optimization:
Selecting order for natural join of three or more relations
Compared to other binary operations, joins take more time and therefore need effective optimization techniques

121. Introduction

122. Significance of Left and Right Join Arguments
The argument relations in joins determine the cost of the join
The left argument of the join is
Called the build relation
Assumed to be smaller
Stored in main-memory

123. Significance of Left and Right Join Arguments
The right argument of the join is
Called the probe relation
Read a block at a time
Its tuples are matched with those of build relation
The join algorithms which distinguish between the arguments are:
One-pass join
Nested-loop join
Index join

124. Join Trees Order of arguments is important for joining two relations
Left argument, since stored in main-memory, should be smaller
With two relations only two choices of join tree
With more than two relations, there are n! ways to order the arguments and therefore n! join trees, where n is the no. of relations

125. Join Trees Order of arguments is important for joining two relations
Left argument, since stored in main-memory, should be smaller
With two relations only two choices of join tree
With more than two relations, there are n! ways to order the arguments and therefore n! join trees, where n is the no. of relations

126. Join Trees
Total # of tree shapes T(n) for n relations given by recurrence:
T(1) = 1
T(2) = 1
T(3) = 2
T(4) = 5 … etc

127. Left-Deep Join Trees
Consider 4 relations. Different ways to join them are as follows

128. In fig (a) all the right children are leaves. This is a left-deep tree
In fig (c) all the left children are leaves. This is a right-deep tree
Fig (b) is a bushy tree
Considering left-deep trees is advantageous for deciding join orders

129. Join order Join order selection A1 A2 A3 .. An Left deep join trees
Dynamic programming
Best plan computed for each subset of relations
Best plan (A1, .., An) = min cost plan of( Best plan(A2, .., An) A Best plan(A1, A3, .., An) A … Best plan(A1, .., An-1)) An An Ai

130. Dynamic Programming to Select a Join Order and Grouping
Three choices to pick an order for the join of many relations are:
Consider all of the relations
Consider a subset
Use a heuristic o pick one
Dynamic programming is used either to consider all or a subset
Construct a table of costs based on relation size
Remember only the minimum entry which will required to proceed

131. Dynamic Programming to Select a Join Order and Grouping

132. Dynamic Programming to Select a Join Order and Grouping

133. Dynamic Programming to Select a Join Order and Grouping

134. Dynamic Programming to Select a Join Order and Grouping

135. A Greedy Algorithm for Selecting a Join Order
It is expensive to use an exhaustive method like dynamic programming
Better approach is to use a join-order heuristic for the query optimization
Greedy algorithm is an example of that
Make one decision at a time about order of join and never backtrack on the decisions once made

136. UPDATES Reducing Cost by Heuristics:
Applies for logical query plan
Estimate cost before and after a transformation
Only choose/apply transformation when cost estimations show beneficial
JOIN TREES:
Order of arguments is important for joining two relations
Left argument, since stored in main-memory, should be smaller

137. Completing the Physical-Query-Plan and Chapter 16 Summary (16.7-16.8)
CS257 Spring 2009
Professor Tsau Lin
Student: Suntorn Sae-Eung
Donavon Norwood

138. Outline 16.7 Completing the Physical-Query-Plan
I. Choosing a Selection Method
II. Choosing a Join Method
III. Pipelining Versus Materialization
IV. Pipelining Unary Operations
V. Pipelining Binary Operations
VI. Notation for Physical Query Plan
VII. Ordering the Physical Operations
16.8 Summary of Chapter 16

139. Before complete Physical-Query-Plan
A query previously has been
Parsed and Preprocessed (16.1)
Converted to Logical Query Plans (16.3)
Estimated the Costs of Operations (16.4)
Determined costs by Cost-Based Plan Selection (16.5)
Weighed costs of join operations by choosing an Order for Joins

140. 16.7 Completing the Physical-Query-Plan
3 topics related to turning LP into a complete physical plan
Choosing of physical implementations such as Selection and Join methods
Decisions regarding to intermediate results (Materialized or Pipelined)
Notation for physical-query-plan operators

141. I. Choosing a Selection Method (A)
Algorithms for each selection operators
1. Can we use an created index on an attribute?
If yes, index-scan. Otherwise table-scan)
2. After retrieve all condition-satisfied tuples in (1), then filter them with the rest selection conditions
σC(R) – we access entire relateion R and see which tuples satisfy condition C
If C was of the form “ Attribute equals constants”
Therefore if C is a constant and we had an index on that attribute then operation σC(R) will be implemented as Aθc
Aθc – A is an attribute with an index, c is a constant and is a comparison operator such as ‘=‘ , ‘<‘ or ‘>’

142. Choosing a Selection Method(A) (cont.)
Recall  Cost of query = # disk I/O’s
How costs for various plans are estimated from σC(R) operation
1. Cost of table-scan algorithm
B(R) if R is clustered
T(R) if R is not clustered
2. Cost of a plan picking an equality term (e.g. a = 10) w/ index-scan
B(R) / V(R, a) clustering index
T(R) / V(R, a) nonclustering index
3. Cost of a plan picking an inequality term (e.g. b < 20) w/ index-scan
B(R) / 3 clustering index
T(R) / 3 nonclustering index

143. Example Selection: σx=1 AND y=2 AND z<5 (R) T(R)=5000, B(R)=200,
- Where parameters of R(x, y, z) are :
T(R)=5000, B(R)=200,
V(R,x)=100, and V(R, y)=500
Relation R is clustered
x, y have nonclustering indexes, only index on z is clustering.

144. Example (cont.) Selection options:
Table-scan  filter x, y, z. Cost is B(R) = 200 since R is clustered.
Use index on x =1  filter on y, z. Cost is 50 since T(R) / V(R, x) is (5000/100) = 50 tuples, index is not clustering.
Use index on y =2  filter on x, z. Cost is 10 since T(R) / V(R, y) is (5000/500) = 10 tuples using nonclustering index.
Index-scan on clustering index w/ z < 5  filter x ,y. Cost is about B(R)/3 = 67

145. Example (cont.) Costs option 1 = 200 option 2 = 50 option 3 = 10 
The lowest Cost is option 3.
Therefore, the preferred physical plan
retrieves all tuples with y = 2
then filters for the rest two conditions (x, z).

146. II. Choosing a Join Method
Determine costs associated with each join algorithms:
1. One-pass join, and nested-loop join devotes enough buffer to joining
2. Sort-join is preferred when attributes are pre-sorted or two or more join on the same attribute such as
(R(a, b) S(a, c)) T(a, d)
- where sorting R and S on a will produce result of R S to be sorted on a and used directly in next join

147. Choosing a Join Method (cont.)
3. Index-join for a join with high chance of using index created on the join attribute such as R(a, b) S(b, c)
4. Hashing join is the best choice for unsorted or non- indexing relations which needs multipass join.

148. III. Pipelining Versus Materialization
Materialization (naïve way)
store (intermediate) result of each operations on disk
Pipelining (more efficient way)
Interleave the execution of several operations, the tuples produced by one operation are passed directly to the operations that used it
store (intermediate) result of each operations on buffer, which is implemented on main memory

149. IV. Pipelining Unary Operations
Unary = a-tuple-at-a-time or full relation
selection and projection are the best candidates for pipelining.
Unary
operation
In buf
Out buf R
Unary
operation
In buf
Out buf
M-1 buffers

150. Pipelining Unary Operations (cont.)
Pipelining Unary Operations are implemented by iterators

151. V. Pipelining Binary Operations
Binary operations : ,  , - , , x
The results of binary operations can also be pipelined.
Use one buffer to pass result to its consumer, one block at a time.
The extended example shows tradeoffs and opportunities

152. Example Consider physical query plan for the expression
(R(w, x) S(x, y)) U(y, z)
Assumption
R occupies 5,000 blocks, S and U each 10,000 blocks.
The intermediate result R S occupies k blocks for some k.
Both joins will be implemented as hash-joins, either one- pass or two-pass depending on k
There are 101 buffers available.

153. Example (cont.) First consider join R S, neither relations
fits in buffers
Needs two-pass
hash-join to partition
R into 100 buckets
(maximum possible) each bucket has 50 blocks
The 2nd pass hash-join uses 51 buffers, leaving the rest 50 buffers for joining result of R S with U.

154. Example (cont.)
Case 1: suppose k  49, the result of R S occupies at most 49 blocks.
Steps
Pipeline in R S into 49 buffers
Organize them for lookup as a hash table
Use one buffer left to read each block of U in turn
Execute the second join as one-pass join.

155. Example (cont.) The total number of I/O’s is 55,000
45,000 for two-pass hash join of R and S
10,000 to read U for one- pass hash join of
(R S) U.

156. Example (cont.)
Case 2: suppose k > 49 but < 5,000, we can still pipeline, but need another strategy which intermediate results join with U in a 50-bucket, two- pass hash-join. Steps are:
Before start on R S, we hash U into 50 buckets of 200 blocks each.
Perform two-pass hash join of R and U using 51 buffers as case 1, and placing results in 50 remaining buffers to form 50 buckets for the join of R S with U.
Finally, join R S with U bucket by bucket.

157. Example (cont.) The number of disk I/O’s is:
20,000 to read U and write its tuples into buckets
45,000 for two-pass hash-join R S
k to write out the buckets of R S
k+10,000 to read the buckets of R S and U in the final join
The total cost is 75,000+2k.

158. Example (cont.) Compare Increasing I/O’s between case 1 and case 2
k  49 (case 1)
Disk I/O’s is 55,000
k > 50  5000 (case 2)
k=50 , I/O’s is 75,000+(2*50) = 75,100
k=51 , I/O’s is 75,000+(2*51) = 75,102
k=52 , I/O’s is 75,000+(2*52) = 75,104
Notice: I/O’s discretely grows as k increases from 49 50.

159. Example (cont.)
Case 3: k > 5,000, we cannot perform two- pass join in 50 buffers available if result of R S is pipelined. Steps are
Compute R S using two-pass join and store the result on disk.
Join result on (1) with U, using two-pass join.

160. Example (cont.) The number of disk I/O’s is:
45,000 for two-pass hash-join R and S
k to store R S on disk
30,000 + k for two-pass join of U in R S
The total cost is 75,000+4k.

161. Example (cont.)
In summary, costs of physical plan as function of R S size.

162. VI. Notation for Physical Query Plans
Several types of operators:
Operators for leaves
(Physical) operators for Selection
(Physical) Sorts Operators
Other Relational-Algebra Operations
In practice, each DBMS uses its own internal notation for physical query plan.

163. Notation for Physical Query Plans (cont.)
Operator for leaves
A leaf operand is replaced in LQP tree
TableScan(R) : read all blocks
SortScan(R, L) : read in order according to L
IndexScan(R, C): scan index attribute A by condition C of form Aθc.
IndexScan(R, A) : scan index attribute R.A. This behaves like TableScan but more efficient if R is not clustered.

164. Notation for Physical Query Plans (cont.)
(Physical) operators for Selection
Logical operator σC(R) is often combined with access methods.
If σC(R) is replaced by Filter(C), and there is no index on R or an attribute on condition C
Use TableScan or SortScan(R, L) to access R
If condition C  Aθc AND D for condition D, and there is an index on R.A, then we may
Use operator IndexScan(R, Aθc) to access R and
Use Filter(D) in place of the selection σC(R)

165. Notation for Physical Query Plans (cont.)
(Physical) Sort Operators
Sorting can occur any point in physical plan, which use a notation SortScan(R, L).
It is common to use an explicit operator Sort(L) to sort relation that is not stored.
Can apply at the top of physical-query-plan tree if the result needs to be sorted with ORDER BY clause (г).

166. Notation for Physical Query Plans (cont.)
Other Relational-Algebra Operations
Descriptive text definitions and signs to elaborate
Operations performed e.g. Join or grouping.
Necessary parameters e.g. theta-join or list of elements in a grouping.
A general strategy for the algorithm e.g. sort-based, hashed based, or index-based.
A decision about number of passed to be used e.g. one- pass, two-pass or multipass.
An anticipated number of buffers the operations will required.

167. Notation for Physical Query Plans (cont.)
Example of a physical-query-plan
A physical-query-plan in example for the case k > 5000
TableScan
Two-pass hash join
Materialize (double line)
Store operator

168. Notation for Physical Query Plans (cont.)
Another example
A physical-query-plan in example for the case k < 49
TableScan
(2) Two-pass hash join
Pipelining
Different buffers needs
Store operator

169. Notation for Physical Query Plans (cont.)
A physical-query-plan in example 16.35
Use Index on condition y = 2 first
Filter with the rest condition later on.

170. VII. Ordering of Physical Operations
The PQP is represented as a tree structure implied order of operations.
Still, the order of evaluation of interior nodes may not always be clear.
Iterators are used in pipeline manner
Overlapped time of various nodes will make “ordering” no sense.

171. Ordering of Physical Operations (cont.)
3 rules summarize the ordering of events in a PQP tree:
Break the tree into sub-trees at each edge that represent materialization.
Execute one subtree at a time.
Order the execution of the subtree
Bottom-top
Left-to-right
All nodes of each sub-tree are executed simultaneously.

172. Summary of Chapter 16
In this part of the presentation I will talk about the main topics of Chapter 16.

173. COMPILATION OF QUERIES
Compilation means turning a query into a physical query plan, which can be implemented by query engine.
Steps of query compilation :
Parsing
Semantic checking
Selection of the preferred logical query plan
Generating the best physical plan

174. THE PARSER The first step of SQL query processing.
Generates a parse tree
Nodes in the parse tree corresponds to the SQL constructs
Similar to the compiler of a programming language

175. VIEW EXPANSION A very critical part of query compilation.
Expands the view references in the query tree to the actual view.
Provides opportunities for the query optimization.

176. SEMANTIC CHECKING Checks the semantics of a SQL query.
Examines a parse tree.
Checks :
Attributes
Relation names
Types
Resolves attribute references.

177. CONVERSION TO A LOGICAL QUERY PLAN
Converts a semantically parsed tree to a algebraic expression.
Conversion is straightforward but sub queries need to be optimized.
Two argument selection approach can be used.

178. ALGEBRAIC TRANSFORMATION
Many different ways to transform a logical query plan to an actual plan using algebraic transformations.
The laws used for this transformation :
Commutative and associative laws
Laws involving selection
Pushing selection
Laws involving projection
Laws about joins and products
Laws involving duplicate eliminations
Laws involving grouping and aggregation

179. ESTIMATING SIZES OF RELATIONS
True running time is taken into consideration when selecting the best logical plan.
Two factors the affects the most in estimating the sizes of relation :
Size of relations ( No. of tuples )
No. of distinct values for each attribute of each relation
Histograms are used by some systems.

180. COST BASED OPTIMIZING
Best physical query plan represents the least costly plan.
Factors that decide the cost of a query plan :
Order and grouping operations like joins, unions and intersections.
Nested loop and the hash loop joins used.
Scanning and sorting operations.
Storing intermediate results.

181. PLAN ENUMERATION STRATEGIES
Common approaches for searching the space for best physical plan .
Dynamic programming : Tabularizing the best plan for each sub expression
Selinger style programming : sort-order the results as a part of table
Greedy approaches : Making a series of locally optimal decisions
Branch-and-bound : Starts with enumerating the worst plans and reach the best plan

182. LEFT-DEEP JOIN TREES
Left – Deep Join Trees are the binary trees with a single spine down the left edge and with leaves as right children.
This strategy reduces the number of plans to be considered for the best physical plan.
Restrict the search to Left – Deep Join Trees when picking a grouping and order for the join of several relations.

183. PHYSICAL PLANS FOR SELECTION
Breaking a selection into an index-scan of relation, followed by a filter operation.
The filter then examines the tuples retrieved by the index-scan.
Allows only those to pass which meet the portions of selection condition.

184. PIPELINING VERSUS MATERIALIZING
This flow of data between the operators can be controlled to implement “ Pipelining “ .
The intermediate results should be removed from main memory to save space for other operators.
This techniques can implemented using “ materialization “ .
Both the pipelining and the materialization should be considered by the physical query plan generator.
An operator always consumes the result of other operator and is passed through the main memory.

185. Pipelining Versus Materialization
UPDATES
Pipelining Versus Materialization
Materialization (naïve way)
store (intermediate) result of each operations on disk
Pipelining (more efficient way)
Interleave the execution of several operations, the tuples produced by one operation are passed directly to the operations that used it
store (intermediate) result of each operations on buffer, which is implemented on main memory