1. 1 Chapter 21 Query Processing Transparencies ownerNoclient © Pearson Education Limited 1995, 2005

2. 2 Chapter 21 - Objectives u Objectives of query processing and optimization. u Static versus dynamic query optimization. u How a query is decomposed and semantically analyzed. u How to create a R.A.T. to represent a query. u Rules of equivalence for RA operations. u How to apply heuristic transformation rules to improve efficiency of a query. © Pearson Education Limited 1995, 2005

3. 3 Chapter 21 - Objectives u Types of database statistics required to estimate cost of operations. u Different strategies for implementing selection. u How to evaluate cost and size of selection. u Different strategies for implementing join. u How to evaluate cost and size of join. u Different strategies for implementing projection. u How to evaluate cost and size of projection. © Pearson Education Limited 1995, 2005

4. 4 Chapter 21 - Objectives u How to evaluate the cost and size of other RA operations. u How pipelining can be used to improve efficiency of queries. u Difference between materialization and pipelining. u Advantages of left-deep trees. u Approaches to finding optimal execution strategy. u How Oracle handles QO. © Pearson Education Limited 1995, 2005

5. 5 Introduction u In network and hierarchical DBMSs, low-level procedural query language is generally embedded in high-level programming language. u Programmer’s responsibility to select most appropriate execution strategy. u With declarative languages such as SQL, user specifies what data is required rather than how it is to be retrieved. u Relieves user of knowing what constitutes good execution strategy. © Pearson Education Limited 1995, 2005

6. 6 Introduction u Also gives DBMS more control over system performance. u Two main techniques for query optimization: –heuristic rules that order operations in a query; –comparing different strategies based on relative costs, and selecting one that minimizes resource usage. u Disk access tends to be dominant cost in query processing for centralized DBMS. © Pearson Education Limited 1995, 2005

7. 7 Query Processing Activities involved in retrieving data from the database. u Aims of QP: –transform query written in high-level language (e.g. SQL), into correct and efficient execution strategy expressed in low-level language (implementing RA); –execute strategy to retrieve required data. © Pearson Education Limited 1995, 2005

8. 8 Query Optimization Activity of choosing an efficient execution strategy for processing query. u As there are many equivalent transformations of same high-level query, aim of QO is to choose one that minimizes resource usage. u Generally, reduce total execution time of query. u May also reduce response time of query. u Problem computationally intractable with large number of relations, so strategy adopted is reduced to finding near optimum solution. © Pearson Education Limited 1995, 2005

9. 9 Example 21.1 - Different Strategies Find all Managers who work at a London branch. SELECT * FROM Staff s, Branch b WHERE s.branchNo = b.branchNo AND (s.position = ‘Manager’ AND b.city = ‘London’); © Pearson Education Limited 1995, 2005

10. 10 Example 21.1 - Different Strategies u Three equivalent RA queries are: (1)  (position='Manager')  (city='London')  (Staff.branchNo=Branch.branchNo) (Staff X Branch) (2)  (position='Manager')  (city='London') ( Staff Staff.branchNo=Branch.branchNo Branch) (3) (  position='Manager' (Staff)) Staff.branchNo=Branch.branchNo (  city='London' (Branch)) © Pearson Education Limited 1995, 2005

11. 11 Example 21.1 - Different Strategies u Assume: –1000 tuples in Staff; 50 tuples in Branch; –50 Managers; 5 London branches; –no indexes or sort keys; –results of any intermediate operations stored on disk; –cost of the final write is ignored; –tuples are accessed one at a time. © Pearson Education Limited 1995, 2005

12. 12 Example 21.1 - Cost Comparison u Cost (in disk accesses) are: (1) (1000 + 50) + 2*(1000 * 50) = 101 050 (2) 2*1000 + (1000 + 50) = 3 050 (3) 1000 + 2*50 + 5 + (50 + 5) = 1 160 u Cartesian product and join operations much more expensive than selection, and third option significantly reduces size of relations being joined together. © Pearson Education Limited 1995, 2005

13. 13 Phases of Query Processing u QP has four main phases: –decomposition (consisting of parsing and validation); –optimization; –code generation; –execution. © Pearson Education Limited 1995, 2005

14. 14 Phases of Query Processing © Pearson Education Limited 1995, 2005

15. 15 Dynamic versus Static Optimization u Two times when first three phases of QP can be carried out: –dynamically every time query is run; –statically when query is first submitted. u Advantages of dynamic QO arise from fact that information is up to date. u Disadvantages are that performance of query is affected, time may limit finding optimum strategy. © Pearson Education Limited 1995, 2005

16. 16 Dynamic versus Static Optimization u Advantages of static QO are removal of runtime overhead, and more time to find optimum strategy. u Disadvantages arise from fact that chosen execution strategy may no longer be optimal when query is run. u Could use a hybrid approach to overcome this. © Pearson Education Limited 1995, 2005

17. 17 Query Decomposition u Aims are to transform high-level query into RA query and check that query is syntactically and semantically correct. u Typical stages are: –analysis, –normalization, –semantic analysis, –simplification, –query restructuring. © Pearson Education Limited 1995, 2005

18. 18 Analysis u Analyze query lexically and syntactically using compiler techniques. u Verify relations and attributes exist. u Verify operations are appropriate for object type. © Pearson Education Limited 1995, 2005

19. 19 Analysis - Example SELECT staff_no FROM Staff WHERE position > 10; u This query would be rejected on two grounds: –staff_no is not defined for Staff relation (should be staffNo). –Comparison ‘>10’ is incompatible with type position, which is variable character string. © Pearson Education Limited 1995, 2005

20. 20 Analysis u Finally, query transformed into some internal representation more suitable for processing. u Some kind of query tree is typically chosen, constructed as follows: –Leaf node created for each base relation. –Non-leaf node created for each intermediate relation produced by RA operation. –Root of tree represents query result. –Sequence is directed from leaves to root. © Pearson Education Limited 1995, 2005

21. 21 Example 21.1 - R.A.T. © Pearson Education Limited 1995, 2005

22. 22 Normalization u Converts query into a normalized form for easier manipulation. u Predicate can be converted into one of two forms: Conjunctive normal form: (position = 'Manager'  salary > 20000)  (branchNo = 'B003') Disjunctive normal form: (position = 'Manager'  branchNo = 'B003' )  (salary > 20000  branchNo = 'B003') © Pearson Education Limited 1995, 2005

23. 23 Semantic Analysis u Rejects normalized queries that are incorrectly formulated or contradictory. u Query is incorrectly formulated if components do not contribute to generation of result. u Query is contradictory if its predicate cannot be satisfied by any tuple. u Algorithms to determine correctness exist only for queries that do not contain disjunction and negation. © Pearson Education Limited 1995, 2005

24. 24 Semantic Analysis u For these queries, could construct: –A relation connection graph. –Normalized attribute connection graph. Relation connection graph Create node for each relation and node for result. Create edges between two nodes that represent a join, and edges between nodes that represent projection. u If not connected, query is incorrectly formulated. © Pearson Education Limited 1995, 2005

25. 25 Semantic Analysis - Normalized Attribute Connection Graph u Create node for each reference to an attribute, or constant 0. u Create directed edge between nodes that represent a join, and directed edge between attribute node and 0 node that represents selection. u Weight edges a  b with value c, if it represents inequality condition (a  b + c); weight edges 0  a with -c, if it represents inequality condition (a  c). u If graph has cycle for which valuation sum is negative, query is contradictory. © Pearson Education Limited 1995, 2005

26. 26 Example 21.2 - Checking Semantic Correctness SELECT p.propertyNo, p.street FROM Client c, Viewing v, PropertyForRent p WHERE c.clientNo = v.clientNo AND c.maxRent >= 500 AND c.prefType = ‘Flat’ AND p.ownerNo = ‘CO93’; u Relation connection graph not fully connected, so query is not correctly formulated. u Have omitted the join condition (v.propertyNo = p.propertyNo). © Pearson Education Limited 1995, 2005

27. 27 Example 21.2 - Checking Semantic Correctness Relation Connection graph Normalized attribute connection graph © Pearson Education Limited 1995, 2005

28. 28 Example 21.2 - Checking Semantic Correctness SELECT p.propertyNo, p.street FROM Client c, Viewing v, PropertyForRent p WHERE c.maxRent > 500 AND c.clientNo = v.clientNo AND v.propertyNo = p.propertyNo AND c.prefType = ‘Flat’ AND c.maxRent < 200; u Normalized attribute connection graph has cycle between nodes c.maxRent and 0 with negative valuation sum, so query is contradictory. © Pearson Education Limited 1995, 2005

29. 29 Simplification –Detects redundant qualifications, –eliminates common sub-expressions, –transforms query to semantically equivalent but more easily and efficiently computed form. u Typically, access restrictions, view definitions, and integrity constraints are considered. u Assuming user has appropriate access privileges, first apply well-known idempotency rules of boolean algebra. © Pearson Education Limited 1995, 2005

30. 30 Transformation Rules for RA Operations Conjunctive Selection operations can cascade into individual Selection operations (and vice versa).  p  q  r (R) =  p (  q (  r (R))) u Sometimes referred to as cascade of Selection.  branchNo='B003'  salary>15000 (Staff) =  branchNo='B003' (  salary>15000 (Staff)) © Pearson Education Limited 1995, 2005

31. 31 Transformation Rules for RA Operations Commutativity of Selection.  p (  q (R)) =  q (  p (R)) u For example:  branchNo='B003' (  salary>15000 (Staff)) =  salary>15000 (  branchNo='B003' (Staff)) © Pearson Education Limited 1995, 2005

32. 32 Transformation Rules for RA Operations In a sequence of Projection operations, only the last in the sequence is required.  L  M …  N (R) =  L (R) u For example:  lName  branchNo, lName (Staff) =  lName (Staff) © Pearson Education Limited 1995, 2005

33. 33 Transformation Rules for RA Operations Commutativity of Selection and Projection. u If predicate p involves only attributes in projection list, Selection and Projection operations commute:  Ai, …, Am (  p (R)) =  p (  Ai, …, Am (R)) where p  {A 1, A 2, …, A m } u For example:  fName, lName (  lName='Beech' (Staff)) =  lName='Beech' (  fName,lName (Staff)) © Pearson Education Limited 1995, 2005

34. 34 Transformation Rules for RA Operations Commutativity of Theta join (and Cartesian product). R p S = S p R R X S = S X R u Rule also applies to Equijoin and Natural join. For example: Staff staff.branchNo=branch.branchNo Branch = Branch staff.branchNo=branch.branchNo Staff © Pearson Education Limited 1995, 2005

35. 35 Transformation Rules for RA Operations Commutativity of Selection and Theta join (or Cartesian product). u If selection predicate involves only attributes of one of join relations, Selection and Join (or Cartesian product) operations commute:  p (R r S) = (  p (R)) r S  p (R X S) = (  p (R)) X S where p  {A 1, A 2, …, A n } © Pearson Education Limited 1995, 2005

36. 36 Transformation Rules for RA Operations u If selection predicate is conjunctive predicate having form (p  q), where p only involves attributes of R, and q only attributes of S, Selection and Theta join operations commute as:  p  q (R r S) = (  p (R)) r (  q (S))  p  q (R X S) = (  p (R)) X (  q (S)) © Pearson Education Limited 1995, 2005

37. 37 Transformation Rules for RA Operations u For example:  position='Manager'  city='London' (Staff Staff.branchNo=Branch.branchNo Branch) = (  position='Manager' (Staff)) Staff.branchNo=Branch.branchNo (  city='London' (Branch)) © Pearson Education Limited 1995, 2005

38. 38 Transformation Rules for RA Operations Commutativity of Projection and Theta join (or Cartesian product). u If projection list is of form L = L 1  L 2, where L 1 only has attributes of R, and L 2 only has attributes of S, provided join condition only contains attributes of L, Projection and Theta join commute:  L1  L2 (R r S) = (  L1 (R)) r (  L2 (S)) © Pearson Education Limited 1995, 2005

39. 39 Transformation Rules for RA Operations u If join condition contains additional attributes not in L (M = M 1  M 2 where M 1 only has attributes of R, and M 2 only has attributes of S), a final projection operation is required:  L1  L2 (R r S) =  L1  L2 ( (  L1  M1 (R)) r (  L2  M2 (S))) © Pearson Education Limited 1995, 2005

40. 40 Transformation Rules for RA Operations u For example:  position,city,branchNo (Staff Staff.branchNo=Branch.branchNo Branch) = (  position, branchNo (Staff)) Staff.branchNo=Branch.branchNo (  city, branchNo (Branch)) u and using the latter rule:  position, city (Staff Staff.branchNo=Branch.branchNo Branch) =  position, city ((  position, branchNo (Staff)) Staff.branchNo=Branch.branchNo (  city, branchNo (Branch))) © Pearson Education Limited 1995, 2005

41. 41 Transformation Rules for RA Operations Commutativity of Union and Intersection (but not set difference). R  S = S  R R  S = S  R © Pearson Education Limited 1995, 2005

42. 42 Transformation Rules for RA Operations Commutativity of Selection and set operations (Union, Intersection, and Set difference).  p (R  S) =  p (S)   p (R)  p (R  S) =  p (S)   p (R)  p (R - S) =  p (S) -  p (R) © Pearson Education Limited 1995, 2005

43. 43 Transformation Rules for RA Operations Commutativity of Projection and Union.  L (R  S) =  L (S)   L (R) Associativity of Union and Intersection (but not Set difference). (R  S)  T = S  (R  T) (R  S)  T = S  (R  T) © Pearson Education Limited 1995, 2005

44. 44 Transformation Rules for RA Operations Associativity of Theta join (and Cartesian product). u Cartesian product and Natural join are always associative: (R S) T = R (S T) (R X S) X T = R X (S X T) u If join condition q involves attributes only from S and T, then Theta join is associative: (R p S) q  r T = R p  r (S q T) © Pearson Education Limited 1995, 2005

45. 45 Transformation Rules for RA Operations u For example: (Staff Staff.staffNo=PropertyForRent.staffNo PropertyForRent) ownerNo=Owner.ownerNo  staff.lName=Owner.lName Owner = Staff staff.staffNo=PropertyForRent.staffNo  staff.lName=lName (PropertyForRent ownerNo Owner) © Pearson Education Limited 1995, 2005

46. 46 Example 21.3 Use of Transformation Rules For prospective renters of flats, find properties that match requirements and owned by CO93. SELECT p.propertyNo, p.street FROM Client c, Viewing v, PropertyForRent p WHERE c.prefType = ‘Flat’ AND c.clientNo = v.clientNo AND v.propertyNo = p.propertyNo AND c.maxRent >= p.rent AND c.prefType = p.type AND p.ownerNo = ‘CO93’; © Pearson Education Limited 1995, 2005

47. 47 Example 21.3 Use of Transformation Rules © Pearson Education Limited 1995, 2005

48. 48 Example 21.3 Use of Transformation Rules © Pearson Education Limited 1995, 2005

49. 49 Example 21.3 Use of Transformation Rules © Pearson Education Limited 1995, 2005

50. 50 Heuristical Processing Strategies u Perform Selection operations as early as possible. –Keep predicates on same relation together. u Combine Cartesian product with subsequent Selection whose predicate represents join condition into a Join operation. u Use associativity of binary operations to rearrange leaf nodes so leaf nodes with most restrictive Selection operations executed first. © Pearson Education Limited 1995, 2005

51. 51 Heuristical Processing Strategies u Perform Projection as early as possible. –Keep projection attributes on same relation together. u Compute common expressions once. –If common expression appears more than once, and result not too large, store result and reuse it when required. –Useful when querying views, as same expression is used to construct view each time. © Pearson Education Limited 1995, 2005

52. 52 Cost Estimation for RA Operations u Many different ways of implementing RA operations. u Aim of QO is to choose most efficient one. u Use formulae that estimate costs for a number of options, and select one with lowest cost. u Consider only cost of disk access, which is usually dominant cost in QP. u Many estimates are based on cardinality of the relation, so need to be able to estimate this. © Pearson Education Limited 1995, 2005

53. 53 Database Statistics u Success of estimation depends on amount and currency of statistical information DBMS holds. u Keeping statistics current can be problematic. u If statistics updated every time tuple is changed, this would impact performance. u DBMS could update statistics on a periodic basis, for example nightly, or whenever the system is idle. © Pearson Education Limited 1995, 2005

54. 54 Typical Statistics for Relation R nTuples(R) - number of tuples in R. bFactor(R) - blocking factor of R. nBlocks(R) - number of blocks required to store R: nBlocks(R) = [nTuples(R)/bFactor(R)] © Pearson Education Limited 1995, 2005

55. 55 Typical Statistics for Attribute A of Relation R nDistinct A (R) - number of distinct values that appear for attribute A in R. min A (R),max A (R) –minimum and maximum possible values for attribute A in R. SC A (R) - selection cardinality of attribute A in R. Average number of tuples that satisfy an equality condition on attribute A. © Pearson Education Limited 1995, 2005

56. 56 Statistics for Multilevel Index I on Attribute A nLevels A (I) - number of levels in I. nLfBlocks A (I) - number of leaf blocks in I. © Pearson Education Limited 1995, 2005

57. 57 Selection Operation u Predicate may be simple or composite. u Number of different implementations, depending on file structure, and whether attribute(s) involved are indexed/hashed. u Main strategies are: –Linear Search (Unordered file, no index). –Binary Search (Ordered file, no index). –Equality on hash key. –Equality condition on primary key. © Pearson Education Limited 1995, 2005

58. 58 Selection Operation –Inequality condition on primary key. –Equality condition on clustering (secondary) index. –Equality condition on a non-clustering (secondary) index. –Inequality condition on a secondary B + -tree index. © Pearson Education Limited 1995, 2005

59. 59 Estimating Cardinality of Selection u Assume attribute values are uniformly distributed within their domain and attributes are independent. nTuples(S) = SC A (R) u For any attribute B  A of S, nDistinct B (S) = nTuples(S)if nTuples(S) < nDistinct B (R)/2 nDistinct B (R)if nTuples(S) > 2*nDistinct B (R) [(nTuples(S) + nDistinct B (R))/3]otherwise © Pearson Education Limited 1995, 2005

60. 60 Linear Search (Ordered File, No Index) u May need to scan each tuple in each block to check whether it satisfies predicate. u For equality condition on key attribute, cost estimate is: [nBlocks(R)/2] u For any other condition, entire file may need to be searched, so more general cost estimate is: nBlocks(R) © Pearson Education Limited 1995, 2005

61. 61 Binary Search (Ordered File, No Index) u If predicate is of form A = x, and file is ordered on key attribute A, cost estimate: [log 2 (nBlocks(R))] u Generally, cost estimate is: [log 2 (nBlocks(R))] + [SC A (R)/bFactor(R)] - 1 u First term represents cost of finding first tuple using binary search. u Expect there to be SC A (R) tuples satisfying predicate. © Pearson Education Limited 1995, 2005

62. 62 Equality of Hash Key u If attribute A is hash key, apply hashing algorithm to calculate target address for tuple. u If there is no overflow, expected cost is 1. u If there is overflow, additional accesses may be necessary. © Pearson Education Limited 1995, 2005

63. 63 Equality Condition on Primary Key u Can use primary index to retrieve single record satisfying condition. u Need to read one more block than number of index accesses, equivalent to number of levels in index, so estimated cost is: nLevels A (I) + 1 © Pearson Education Limited 1995, 2005

64. 64 Inequality Condition on Primary Key u Can first use index to locate record satisfying predicate (A = x). u Provided index is sorted, records can be found by accessing all records before/after this one. u Assuming uniform distribution, would expect half the records to satisfy inequality, so estimated cost is: nLevels A (I) + [nBlocks(R)/2] © Pearson Education Limited 1995, 2005

65. 65 Equality Condition on Clustering Index u Can use index to retrieve required records. u Estimated cost is: nLevels A (I) + [SC A (R)/bFactor(R)] u Second term is estimate of number of blocks that will be required to store number of tuples that satisfy equality condition, represented as SC A (R). © Pearson Education Limited 1995, 2005

66. 66 Equality Condition on Non-Clustering Index u Can use index to retrieve required records. u Have to assume that tuples are on different blocks (index is not clustered this time), so estimated cost becomes: nLevels A (I) + [SC A (R)] © Pearson Education Limited 1995, 2005

67. 67 Inequality Condition on a Secondary B + - Tree Index u From leaf nodes of tree, can scan keys from smallest value up to x ( or >=). u Assuming uniform distribution, would expect half the leaf node blocks to be accessed and, via index, half the file records to be accessed. u Estimated cost is: nLevels A (I) + [nLfBlocks A (I)/2 + nTuples(R)/2] © Pearson Education Limited 1995, 2005

68. 68 Composite Predicates - Conjunction without Disjunction u May consider following approaches: - If one attribute has index or is ordered, can use one of above selection strategies. Can then check each retrieved record. - For equality on two or more attributes, with composite index (or hash key) on combined attributes, can search index directly. - With secondary indexes on one or more attributes (involved only in equality conditions in predicate), could use record pointers if exist. © Pearson Education Limited 1995, 2005

69. 69 Composite Predicates - Selections with Disjunction u If one term contains an  (OR), and term requires linear search, entire selection requires linear search. u Only if index or sort order exists on every term can selection be optimized by retrieving records that satisfy each condition and applying union operator. u Again, record pointers can be used if they exist. © Pearson Education Limited 1995, 2005

70. 70 Join Operation u Main strategies for implementing join: –Block Nested Loop Join. –Indexed Nested Loop Join. –Sort-Merge Join. –Hash Join. © Pearson Education Limited 1995, 2005

71. 71 Estimating Cardinality of Join u Cardinality of Cartesian product is: nTuples(R) * nTuples(S) u More difficult to estimate cardinality of any join as depends on distribution of values. u Worst case, cannot be any greater than this value. © Pearson Education Limited 1995, 2005

72. 72 Estimating Cardinality of Join u If assume uniform distribution, can estimate for Equijoins with a predicate (R.A = S.B) as follows: –If A is key of R: nTuples(T)  nTuples(S) –If B is key of S: nTuples(T)  nTuples(R) u Otherwise, could estimate cardinality of join as: nTuples(T) = SC A (R)*nTuples(S) or nTuples(T) = SC B (S)*nTuples(R) © Pearson Education Limited 1995, 2005

73. 73 Block Nested Loop Join u Simplest join algorithm is nested loop that joins two relations together a tuple at a time. u Outer loop iterates over each tuple in R, and inner loop iterates over each tuple in S. u As basic unit of reading/writing is a disk block, better to have two extra loops that process blocks. u Estimated cost of this approach is: nBlocks(R) + (nBlocks(R) * nBlocks(S)) © Pearson Education Limited 1995, 2005

74. 74 Block Nested Loop Join u Could read as many blocks as possible of smaller relation, R say, into database buffer, saving one block for inner relation and one for result. u New cost estimate becomes: nBlocks(R) + [nBlocks(S)*(nBlocks(R)/(nBuffer-2))] u If can read all blocks of R into the buffer, this reduces to: nBlocks(R) + nBlocks(S) © Pearson Education Limited 1995, 2005

75. 75 Indexed Nested Loop Join u If have index (or hash function) on join attributes of inner relation, can use index lookup. u For each tuple in R, use index to retrieve matching tuples of S. u Cost of scanning R is nBlocks(R), as before. u Cost of retrieving matching tuples in S depends on type of index and number of matching tuples. u If join attribute A in S is PK, cost estimate is: nBlocks(R) + nTuples(R)*(nlevels A (I) + 1) © Pearson Education Limited 1995, 2005

76. 76 Sort-Merge Join u For Equijoins, most efficient join is when both relations are sorted on join attributes. u Can look for qualifying tuples merging relations. u May need to sort relations first. u Now tuples with same join value are in order. u If assume join is *:* and each set of tuples with same join value can be held in database buffer at same time, then each block of each relation need only be read once. © Pearson Education Limited 1995, 2005

77. 77 Sort-Merge Join u Cost estimate for the sort-merge join is: nBlocks(R) + nBlocks(S) u If a relation has to be sorted, R say, add: nBlocks(R)*[log 2 (nBlocks(R)] © Pearson Education Limited 1995, 2005

78. 78 Hash Join u For Natural or Equijoin, hash join may be used. u Idea is to partition relations according to some hash function that provides uniformity and randomness. u Each equivalent partition should hold same value for join attributes, although it may hold more than one value. u Cost estimate of hash join as: 3(nBlocks(R) + nBlocks(S)) © Pearson Education Limited 1995, 2005

79. 79 Projection Operation u To implement projection need to: –remove attributes that are not required; –eliminate any duplicate tuples produced from previous step. Only required if projection attributes do not include a key. u Two main approaches to eliminating duplicates: –sorting; –hashing. © Pearson Education Limited 1995, 2005

80. 80 Estimating Cardinality of Projection u When projection contains key, cardinality is: nTuples(S) = nTuples(R) u If projection consists of a single non-key attribute, estimate is: nTuples(S) = SC A (R) u Otherwise, could estimate cardinality as: nTuples(S)  min(nTuples(R),  i m =1 (nDistinct ai (R))) © Pearson Education Limited 1995, 2005

81. 81 Duplicate Elimination using Sorting u Sort tuples of reduced relation using all remaining attributes as sort key. u Duplicates will now be adjacent and can be removed easily. u Estimated cost of sorting is: nBlocks(R)*[log 2 (nBlocks(R))]. u Combined cost is: nBlocks(R) + nBlocks(R)*[log 2 (nBlocks(R))] © Pearson Education Limited 1995, 2005

82. 82 Duplicate Elimination using Hashing u Two phases: partitioning and duplicate elimination. u In partitioning phase, for each tuple in R, remove unwanted attributes and apply hash function to combination of remaining attributes, and write reduced tuple to hashed value. u Two tuples that belong to different partitions are guaranteed not to be duplicates. u Estimated cost is: nBlocks(R) + nB © Pearson Education Limited 1995, 2005

83. 83 Set Operations u Can be implemented by sorting both relations on same attributes, and scanning through each of sorted relations once to obtain desired result. u Could use sort-merge join as basis. u Estimated cost in all cases is: nBlocks(R) + nBlocks(S) + nBlocks(R)*[log 2 (nBlocks(R))] + nBlocks(S)*[log 2 (nBlocks(S))] u Could also use hashing algorithm. © Pearson Education Limited 1995, 2005

84. 84 Estimating Cardinality of Set Operations u As duplicates are eliminated when performing Union, difficult to estimate cardinality, but can give an upper and lower bound as: max(nTuples(R), nTuples(S))  nTuples(T)  nTuples(R) + nTuples(S) u For Set Difference, can also give upper and lower bound: 0  nTuples(T)  nTuples(R) © Pearson Education Limited 1995, 2005

85. 85 Aggregate Operations SELECT AVG(salary) FROM Staff; u To implement query, could scan entire Staff relation and maintain running count of number of tuples read and sum of all salaries. u Easy to compute average from these two running counts. © Pearson Education Limited 1995, 2005

86. 86 Aggregate Operations SELECT AVG(salary) FROM Staff GROUP BY branchNo; u For grouping queries, can use sorting or hashing algorithms similar to duplicate elimination. u Can estimate cardinality of result using estimates derived earlier for selection. © Pearson Education Limited 1995, 2005

87. 87 Enumeration of Alternative Strategies u Fundamental to efficiency of QO is the search space of possible execution strategies and the enumeration algorithm used to search this space. u Query with 2 joins gives 12 join orderings: R (S T) R (T S)(S T) R(T S) R S (R T) S (T R)(R T) S(T R) S T (R S) T (S R)(R S) T(S R) T u With n relations, (2(n – 1))!/(n – 1)! orderings. u If n = 4 this is 120; if n = 10 this is > 176 billion. u Compounded by different selection/join methods. © Pearson Education Limited 1995, 2005

88. 88 Pipelining u Materialization - output of one operation is stored in temporary relation for processing by next. u Could also pipeline results of one operation to another without creating temporary relation. u Known as pipelining or on-the-fly processing. u Pipelining can save on cost of creating temporary relations and reading results back in again. u Generally, pipeline is implemented as separate process or thread. © Pearson Education Limited 1995, 2005

89. 89 Types of Trees © Pearson Education Limited 1995, 2005

90. 90 Pipelining u With linear trees, relation on one side of each operator is always a base relation. u However, as need to examine entire inner relation for each tuple of outer relation, inner relations must always be materialized. u This makes left-deep trees appealing as inner relations are always base relations. u Reduces search space for optimum strategy, and allows QO to use dynamic processing. u Not all execution strategies are considered. © Pearson Education Limited 1995, 2005

91. 91 Physical Operators & Strategies u Term physical operator refers to specific algorithm that implements a logical operation, such as selection or join. u For example, can use sort-merge join to implement the join operation. u Replacing logical operations in a R.A.T. with physical operators produces an execution strategy (or query evaluation plan or access plan). © Pearson Education Limited 1995, 2005

92. 92 Physical Operators & Strategies © Pearson Education Limited 1995, 2005

93. 93 Reducing the Search Space u Restriction 1:Unary operations processed on- the-fly: selections processed as relations are accessed for first time; projections processed as results of other operations are generated. u Restriction 2:Cartesian products are never formed unless query itself specifies one. u Restriction 3:Inner operand of each join is a base relation, never an intermediate result. This uses fact that with left-deep trees inner operand is a base relation and so already materialized. u Restriction 3 excludes many alternative strategies but significantly reduces number to be considered. © Pearson Education Limited 1995, 2005

94. 94 Dynamic Programming u Enumeration of left-deep trees using dynamic programming first proposed for System R QO. u Algorithm based on assumption that the cost model satisfies principle of optimality. u Thus, to obtain optimal strategy for query with n joins, only need to consider optimal strategies for subexpressions with (n – 1) joins and extend those strategies with an additional join. Remaining suboptimal strategies can be discarded. © Pearson Education Limited 1995, 2005

95. 95 Dynamic Programming u To ensure some potentially useful strategies are not discarded algorithm retains strategies with interesting orders: an intermediate result has an interesting order if it is sorted by a final ORDER BY attribute, GROUP BY attribute, or any attributes that participate in subsequent joins. © Pearson Education Limited 1995, 2005

96. 96 Dynamic Programming SELECT p.propertyNo, p.street FROM Client c, Viewing v, PropertyForRent p WHEREc.maxRent < 500 AND c.clientNo = v.clientNo AND v.propertyNo = p.propertyNo; u Attributes c.clientNo, v.clientNo, v.propertyNo, and p.propertyNo are interesting. u If any intermediate result is sorted on any of these attributes, then corresponding partial strategy must be included in search. © Pearson Education Limited 1995, 2005

97. 97 Dynamic Programming u Algorithm proceeds from the bottom up and constructs all alternative join trees that satisfy the restrictions above, as follows: u Pass 1: Enumerate the strategies for each base relation using a linear search and all available indexes on the relation. These partial strategies are partitioned into equivalence classes based on any interesting orders. An additional equivalence class is created for the partial strategies with no interesting order. © Pearson Education Limited 1995, 2005

98. 98 Dynamic Programming u For each equivalence class, strategy with lowest cost is retained for consideration in next pass. u Do not retain equivalence class with no interesting order if its lowest cost strategy is not lower than all other strategies. u For a given relation R, any selections involving only attributes of R are processed on-the-fly. Similarly, any attributes of R that are not part of the SELECT clause and do not contribute to any subsequent join can be projected out at this stage (restriction 1 above). © Pearson Education Limited 1995, 2005

99. 99 Dynamic Programming u Pass 2: Generate all 2-relation strategies by considering each strategy retained after Pass 1 as outer relation, discarding any Cartesian products generated (restriction 2 above). Again, any on-the- fly processing is performed and lowest cost strategy in each equivalence class is retained. u Pass n: Generate all n-relation strategies by considering each strategy retained after Pass (n – 1) as outer relation, discarding any Cartesian products generated. After pruning, now have lowest overall strategy for processing the query. © Pearson Education Limited 1995, 2005

100. 100 Dynamic Programming u Although algorithm is still exponential, there are query forms for which it only generates O(n 3 ) strategies, so for n = 10 the number is 1,000, which is significantly better than the 176 billion different join orders noted earlier. © Pearson Education Limited 1995, 2005

101. 101 Semantic Query Optimization u Based on constraints specified on the database schema to reduce the search space. u For example, a constraint states that staff cannot supervise more than 100 properties, so any query searching for staff who supervise more than 100 properties will produce zero rows. Now consider: CREATE ASSERTION ManagerSalary CHECK (salary > 20000 AND position = ‘Manager’) SELECT s.staffNo, fName, lName, propertyNo FROM Staff s, PropertyForRent p WHEREs.staffNo = p.staffNo AND position = ‘Manager’; © Pearson Education Limited 1995, 2005

102. 102 Semantic Query Optimization u Can rewrite this query as: SELECT s.staffNo, fName, lName, propertyNo FROM Staff s, PropertyForRent p WHEREs.staffNo = p.staffNo AND salary > 20000 AND position = ‘Manager’; u Additional predicate may be very useful if only index for Staff is a B+-tree on the salary attribute. u However, additional predicate would complicate query if no such index existed. © Pearson Education Limited 1995, 2005

103. 103 Query Optimization in Oracle u Oracle supports two approaches to query optimization: rule-based and cost-based. Rule-based u 15 rules, ranked in order of efficiency. Particular access path for a table only chosen if statement contains a predicate or other construct that makes that access path available. u Score assigned to each execution strategy using these rankings and strategy with best (lowest) score selected. © Pearson Education Limited 1995, 2005

104. 104 QO in Oracle – Rule-Based u When 2 strategies have same score, tie-break resolved by making decision based on order in which tables occur in the SQL statement. © Pearson Education Limited 1995, 2005

105. 105 QO in Oracle – Rule-based: Example SELECT propertyNo FROM PropertyForRent WHERE rooms > 7 AND city = ‘London’ u Single-column access path using index on city from WHERE condition (city = ‘London’). Rank 9. u Unbounded range scan using index on rooms from WHERE condition (rooms > 7). Rank 11. u Full table scan - rank 15. u Although there is index on propertyNo, column does not appear in WHERE clause and so is not considered by optimizer. u Based on these paths, rule-based optimizer will choose to use index based on city column. © Pearson Education Limited 1995, 2005

106. 106 QO in Oracle – Cost-Based u To improve QO, Oracle introduced cost-based optimizer in Oracle 7, which selects strategy that requires minimal resource use necessary to process all rows accessed by query (avoiding above tie-break anomaly). u User can select whether minimal resource usage is based on throughput or based on response time, by setting the OPTIMIZER_MODE initialization parameter. u Cost-based optimizer also takes into consideration hints that the user may provide. © Pearson Education Limited 1995, 2005

107. 107 QO in Oracle – Statistics u Cost-based optimizer depends on statistics for all tables, clusters, and indexes accessed by query. u Users’ responsibility to generate these statistics and keep them current. u Package DBMS_STATS can be used to generate and manage statistics. u Whenever possible, Oracle uses a parallel method to gather statistics, although index statistics are collected serially. EXECUTE DBMS_STATS.GATHER_SCHEMA_STATS(‘Manager’); © Pearson Education Limited 1995, 2005

108. 108 QO in Oracle – Histograms u Previously made assumption that data values within columns of a table are uniformly distributed. u Histogram of values and their relative frequencies gives optimizer improved selectivity estimates in presence of non-uniform distribution. © Pearson Education Limited 1995, 2005

109. 109 QO in Oracle – Histograms u (a) uniform distribution of rooms; (b) actual non-uniform distribution. u (a) can be stored compactly as low value (1) and high value (10), and as total count of all frequencies (in this case, 100). © Pearson Education Limited 1995, 2005

110. 110 QO in Oracle – Histograms u Histogram is data structure that can improve estimates of number of tuples in result. u Two types of histogram: –width-balanced histogram, which divides data into a fixed number of equal-width ranges (called buckets) each containing count of number of values falling within that bucket; –height-balanced histogram, which places approximately same number of values in each bucket so that end points of each bucket are determined by how many values are in that bucket. © Pearson Education Limited 1995, 2005

111. 111 QO in Oracle – Histograms u (a) width-balanced for rooms with 5 buckets. Each bucket of equal width with 2 values (1-2, 3-4, etc.) u (b) height-balanced – height of each column is 20 (100/5). © Pearson Education Limited 1995, 2005

112. 112 QO in Oracle – Viewing Execution Plan © Pearson Education Limited 1995, 2005