1. Oracle University SQL Masterclass Rob van WijkJune 9 & 10, 2011, Tallinn, Estonia

2. About me Work with Oracle and SQL since 1995 From: Blog: Forums: Utrecht, Netherlands

3. Agenda Day 1 Part One:Do More With SQL and Joining Part Two:Analytic Functions Part Three:Grouping & Aggregating Part Four:SQL Model Clause Day 2 Part Five:Recursive Subquery Factoring Part Six:Regular Expressions Part Seven:XML in SQL Part Eight:Frequently Occuring SQL Problems

4. Part 1a: Do More With SQL

5. Goals As practical as possible As less regurgitating of documentation as possible Lots of example scripts Recognizable problems Do as much as possible in SQL and avoid shipping records for processing to PL/SQL or even Java at a middle tier.

6. If you want to build a ship, don't drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea. – Antoine de Saint Exupéry

8. One SQL engine versus …

9. … two engines. procedural engine SQL engine context swtiches dmws1.sql

10. You risk wrong results because of different start times of queries with default READ COMMITTED isolation level dmws2.sql

11. Part 1b: ANSI joins and partitioned outer join

12. ANSI joins: Comparison with Oracle-syntax CROSS JOIN INNER JOIN OUTER JOIN NATURAL JOIN aj3.sql aj2.sql aj1.sql aj4.sql

13. ANSI joins: Full Outer Join a FULL OUTER JOIN b ≡ a LEFT OUTER JOIN b UNION ALL b WHERE NOT EXISTS a 11g: Native full outer join _optimizer_native_full_outer_join /*+ NATIVE_FULL_OUTER_JOIN */ /*+ NO_NATIVE_FULL_OUTER_JOIN */ aj5.sql aj6.sql

14. ANSI joins: Partitioned Outer Join Outer join: NULL rows for missing values Partitioned outer join: NULL rows for missing values per, …, aj7.sql

15. ANSI joins 15Title of presentation

16. Part 2: Analytic Functions

17. Analytic Functions: Topics Introduction Mind set Evaluation order Main syntax Examples Window clause

18. Analytic Functions: Introduction Of every employee please show me: His name The department he’s working in His salary The cumulative salary per department Percentage of salary within the department Percentage of salary within the company where employees are sorted by department and salary af1c.sql af1b.sql af1a.sql

19. Analytic Functions: Introduction Since 8.1.6 Enterprise Edition Look like well known aggregate functions like SUM, COUNT and AVG … but they don’t aggregate Prevents self joins Have been extended with new functions and new options in more recent versions af2.sql

20. Analytic Functions: Mind set Don’t think “rows” … EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO ---------- ---------- --------- ---------- -------- ---------- ---------- ---------- 7782 CLARK MANAGER 7839 09-06-81 2450 10 7839 KING PRESIDENT 17-11-81 5000 10 7934 MILLER CLERK 7782 23-01-82 1300 10 7876 ADAMS CLERK 7788 23-05-87 1100 20 7902 FORD ANALYST 7566 03-12-81 3000 20 7566 JONES MANAGER 7839 02-04-81 2975 20 7788 SCOTT ANALYST 7566 19-04-87 3000 20 7369 SMITH CLERK 7902 17-12-80 800 20 7499 ALLEN SALESMAN 7698 20-02-81 1600 300 30 7698 BLAKE MANAGER 7839 01-05-81 2850 30 7900 JAMES CLERK 7698 03-12-81 950 30 7654 MARTIN SALESMAN 7698 28-09-81 1250 1400 30 7844 TURNER SALESMAN 7698 08-09-81 1500 0 30 7521 WARD SALESMAN 7698 22-02-81 1250 500 30 7902 FORD ANALYST 7566 03-12-81 3000 20

21. Analytic Functions: Mind set … but think “sets” EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO ---------- ---------- --------- ---------- -------- ---------- ---------- ---------- 7782 CLARK MANAGER 7839 09-06-81 2450 10 7839 KING PRESIDENT 17-11-81 5000 10 7934 MILLER CLERK 7782 23-01-82 1300 10 7876 ADAMS CLERK 7788 23-05-87 1100 20 7902 FORD ANALYST 7566 03-12-81 3000 20 7566 JONES MANAGER 7839 02-04-81 2975 20 7788 SCOTT ANALYST 7566 19-04-87 3000 20 7369 SMITH CLERK 7902 17-12-80 800 20 7499 ALLEN SALESMAN 7698 20-02-81 1600 300 30 7698 BLAKE MANAGER 7839 01-05-81 2850 30 7900 JAMES CLERK 7698 03-12-81 950 30 7654 MARTIN SALESMAN 7698 28-09-81 1250 1400 30 7844 TURNER SALESMAN 7698 08-09-81 1500 0 30 7521 WARD SALESMAN 7698 22-02-81 1250 500 30 EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO ---------- ---------- --------- ---------- -------- ---------- ---------- ---------- 7782 CLARK MANAGER 7839 09-06-81 2450 10 7839 KING PRESIDENT 17-11-81 5000 10 7934 MILLER CLERK 7782 23-01-82 1300 10 7876 ADAMS CLERK 7788 23-05-87 1100 20 7902 FORD ANALYST 7566 03-12-81 3000 20 7566 JONES MANAGER 7839 02-04-81 2975 20 7788 SCOTT ANALYST 7566 19-04-87 3000 20 7369 SMITH CLERK 7902 17-12-80 800 20 7499 ALLEN SALESMAN 7698 20-02-81 1600 300 30 7698 BLAKE MANAGER 7839 01-05-81 2850 30 7900 JAMES CLERK 7698 03-12-81 950 30 7654 MARTIN SALESMAN 7698 28-09-81 1250 1400 30 7844 TURNER SALESMAN 7698 08-09-81 1500 0 30 7521 WARD SALESMAN 7698 22-02-81 1250 500 30

22. Analytic Functions: Evaluation order Last Even after evaluating HAVING clause And after ROWNUM has been assigned But before ORDER BY clause Filtering on outcome of analytic function: nest the query using an inline view or use subquery factoring af3b.sql af3a.sql

23. Analytic Functions: Main syntax (,, …) OVER ( )

24. Analytic Functions: The functions LAGFIRST / LASTPERCENT_RANK LEADCOUNTPERCENTILE_DISC FIRST_VALUESUMPERCENTILE_CONT LAST_VALUEMAXCORR NTH_VALUEMINCOVAR_POP RANKAVGVARIANCE DENSE_RANKNTILEVAR_x (2 times) RATIO_TO_REPORTCUME_DISTSTDDEV_x (3 times) ROW_NUMBERLISTAGGREGR_x (9 times)

25. Analytic Functions: Partition clause PARTITION BY [, ]* to let the analytic function operate on a subset of the rows with the same values for the partition by expression values. af4.sql

26. Analytic Functions: Order By clause ORDER BY [ASC|DESC] [NULLS FIRST|NULLS LAST], … Its presence changes the default window of an analytic function from the total set to a running total. af5.sql

27. Analytic Functions: Example 1 Top N queries What do I mean exactly with: “Show me the top 3 earning employees per department” RANK DENSE_RANK ROW_NUMBER af6.sql

28. Analytic Functions: Example 2 1. David Zabriskie (USA)0.58:31 2. Ivan Basso (ITA) + 0:17 3. Paolo Savoldelli (ITA) + 0:44 4. Marzio Bruseghin (ITA) + 0:48 5. Serguei Gonchar (UKR) z.t. 6. Vladimir Karpets (RUS) + 1:07 7. Markus Fothen (GER) + 1:15 8. Thomas Dekker (NLD) + 1:23 9. Jan Hruska (CZE) + 1:34 10. Danilo di Luca (ITA) z.t. af7.sql

29. Analytic Functions: Example 3 Requirement: non-overlapping & consecutive periods Columns Startdate and maybe Enddate Optimize to retrieve current period Options: 1)No Enddate column and use correlated subquery 2)Enddate column and database trigger code to check requirement 3)No Enddate column and use analytic function af8.sql

30. Analytic Functions: Example 4 Bills can be of type “Prepayment” or “Settlement” Bill lines have an amount. Each customer pays a prepayment each month. The bill contains one bill line with the amount. Each customer receives once a year a settlement bill. How to calculate the previous prepayment amount? This is the amount before the last settlement bill. af9.sql

31. Analytic Functions: Example 5 TIME QUANTITY -------- ----------- 12:22:01 100 12:22:03 200 12:22:04 300 12:22:06 200 12:22:45 100 12:22:46 200 12:23:12 100 12:23:12 200 MIN(TIME) MAX(TIME) QUANTITY --------- --------- ----------- 12:22:01 12:22:06 800 12:22:45 12:22:46 300 12:23:12 12:23:12 300 af10.sql

32. Analytic Functions: Window clause Total set: ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING Anchored set / running aggregate: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ROW / RANGE af13.sql af11.sql af12.sql

33. Analytic Functions 33Title of presentation

34. Part 3: Grouping & Aggregating

35. aog1.sql

36. Grouping & Aggregating: Topics Introduction GROUPING SETS ROLLUP CUBE Combining and calculating Supporting functions Inner workings MIN/MAX … KEEP … (DENSE_RANK FIRST/LAST … )

37. Grouping & Aggregating: Grouping Sets (1) GROUP BY expr 1, …, expr n ≡ GROUP BY GROUPING SETS ( (expr 1, …, expr n ) ) aog2.sql

38. Grouping & Aggregating: Grouping Sets (2) GROUP BY GROUPING SETS ( (expr 11, …, expr 1n ), …, (expr x1, …, expr xm ) ) ≡ GROUP BY expr 11, … expr 1n UNION ALL … UNION ALL GROUP BY expr x1, …, expr xm aog3.sql

39. Grouping & Aggregating: ROLLUP (1) GROUP BY ROLLUP ( set 1, …, set n ) ≡ GROUP BY GROUPING SETS ( (set 1, …, set n ), (set 1, …, set n-1 ), …, set 1, () )

40. Grouping & Aggregating: ROLLUP (2) ROLLUP (set 1, …, set N ) with N ≥ 1 leads to N+1 GROUPING SETS

41. Grouping & Aggregating: ROLLUP (3) Example: GROUP BY ROLLUP ( (deptno), (job,mgr), (empno) ) ≡ GROUP BY GROUPING SETS ( (deptno,job,mgr,empno), (deptno,job,mgr), (deptno), () ) aog4.sql

42. Grouping & Aggregating: CUBE (1) GROUP BY CUBE ( set 1, …, set n ) ≡ GROUP BY GROUPING SETS (all possible combinations between () and (set 1, …, set n ) )

43. Grouping & Aggregating: CUBE (2) CUBE (set 1, …, set N ) with N ≥ 1 leads to 2 N GROUPING SETS

44. Grouping & Aggregating: CUBE (3) Follows Pascal’s triangle 0 sets X 1 set 2 sets 3 sets 4 sets

45. Grouping & Aggregating: CUBE (4) Example: GROUP BY CUBE ( (deptno), (job,mgr), (empno) ) ≡ GROUP BY GROUPING SETS ( (deptno,job,mgr,empno), (deptno,job,mgr), (deptno,empno), (job,mgr,empno), (deptno), (job,mgr), (empno), () ) aog5.sql

46. Grouping & Aggregating: Calculating (1) GROUP BY deptno, ROLLUP(empno) ?

47. Grouping & Aggregating: Calculating (2) GROUP BY deptno, ROLLUP(empno) ≡ GROUP BY GROUPING SETS (deptno), GROUPING SETS ( empno, () )

48. Grouping & Aggregating: Calculating (3) Cartesian product ! GROUP BY deptno, ROLLUP(empno) ≡ GROUP BY GROUPING SETS (deptno), GROUPING SETS ( (empno), () ) ≡ GROUP BY GROUPING SETS ( (deptno,empno), (deptno) ) aog6.sql

49. Grouping & Aggregating: Calculating (4) Question: How many grouping sets does the clause below yield? GROUP BY ROLLUP(deptno,job), CUBE(mgr,hiredate) aog7.sql

50. Grouping & Aggregating: Functions GROUPING GROUPING_ID GROUP_ID aog8.sql

51. Grouping & Aggregating: Inner working (1) SORT GROUP BY Versus HASH GROUP BY

52. Grouping & Aggregating: Inner working (2) 10 7782 2450 10 7839 5000 10 7934 1300 20 7369 800 20 7566 2975 20 7788 3000 20 7876 1100 20 7902 3000 30 7499 1600 30 7521 1250 30 7654 1250 30 7698 2850 30 7844 1500 30 7900 950 10 NULL 8750 20 NULL 10875 30 NULL 9400 + + NULL 29025 incoming set grouping set ( (deptno,empno) ) grouping set ( () ) grouping set ( (deptno) ) SORT GROUP BY ROLLUP (DEPTNO,EMPNO) aog9.sql

53. Grouping & Aggregating: Inner working (3) SORT GROUP BY (deptno,job) GENERATE CUBE SORT GROUP BY (deptno,job) incoming set 14 rows 9 rows 36 rows 18 rows CUBE (DEPTNO,JOB) aog10.sql deptno null & job null deptno not null & job not nulldeptno not null & job null deptno null & job not null

54. Grouping & Aggregating: Inner working (4) LOAD AS SELECT (into input table) TABLE ACCESS FULL (EMP) TEMP TABLE TRANSFORMATION VIEW TABLE ACCESS FULL (output table) temporary input table SYS_TEMP_... temporary output table SYS_TEMP_... LOAD AS SELECT (into outputtable) HASH GROUP BY TABLE ACCESS FULL (input table) iterate as much times as there are grouping sets aog11.sql

55. Grouping & Aggregating: Inner working (5) Optimize towards a ROLLUP or CUBE execution, if possible? aog12.sql

56. Grouping & Aggregating: Agg. Functions (1) COUNT SUM AVG MAX MIN STDDEV VARIANCE LISTAGG aog13.sql

57. Grouping & Aggregating: Agg. Functions (2) MAX(…) KEEP (DENSE_RANK FIRST ORDER BY …) MAX(…) KEEP (DENSE_RANK LAST ORDER BY …) MIN(…) KEEP (DENSE_RANK FIRST ORDER BY …) MIN(…) KEEP (DENSE_RANK LAST ORDER BY …) aog14.sql

58. Grouping & Aggregating 58Title of presentation

59. Part 4: SQL Model Clause

60. SQL Model Clause: Topics Introduction Syntax Examples Performance Alternatives Conclusion

61. SQL Model Clause: Introduction Treat data as multidimensional arrays Complex calculations across rows Syntax which resembles logic programming (Prolog) Can prevent exporting data to external applications like Excel/Numbers No more several copies of data on several PC’s anymore

62. SQL Model Clause: Syntax (1) Divide columns in three groups: PARTITION, DIMENSION and MEASURES Every partition is a separate array Dimensions identify a cell in every partition Measures are the columns you want to (re-)calculate The rules tell you how the data is to be manipulated

63. SQL Model Clause: Examples A model clause that does nothing Adding an extra row to the result set RETURN UPDATED ROWS The difference between MEASURES and PARTITION mc4.sql mc3.sql mc2.sql mc1.sql

64. SQL Model Clause: Example from the doc mc5.sql

65. SQL Model Clause: More examples ANY CV() FOR Iterating Reference models Difference between NULL and NAV IS PRESENT, PRESENTV and PRESENTNNV mc11.sql mc10.sql mc9.sql mc8.sql mc7.sql mc6.sql mc12.sql

66. SQL Model Clause: Complete syntax MODEL [ ] [MAIN ] [PARTITION BY ( )] DIMENSION BY ( ) MEASURES ( ) [ ] [RULES] (,,.., ) ::= ::= RETURN {ALL|UPDATED} ROWS ::= [IGNORE NAV | [KEEP NAV] [UNIQUE DIMENSION | UNIQUE SINGLE REFERENCE] ::= [UPDATE | UPSERT | UPSERT ALL] [AUTOMATIC ORDER | SEQUENTIAL ORDER] [ITERATE ( ) [UNTIL ]] ::= REFERENCE ON ON ( ) DIMENSION BY ( ) MEASURES ( )

67. SQL Model Clause: Examples Financial spreadsheet Fibonacci OTN-question Interest and rates mc13.sql mc14.sql mc15.sql mc16.sql

68. SQL Model Clause: Performance Internal hash-tables in PGA Sequential Order  SQL MODEL ORDERED [FAST] Automatic Order  SQL MODEL [A]CYCLIC FAST  left side cell references are single cell references and aggregates at right side -if any- are simple arithmetic non-distinct aggregates, like SUM, COUNT, AVG and so on. mc17.sql

69. SQL Model Clause: Advanced examples Calculating die probabilities Exponential Moving Average X = (K * (C - P)) + P Where: X = Current EMA (i.e. EMA to be calculated) C = Current original data value K = Smoothing Constant P = Previous EMA Sudoku solver mc18.sql mc19.sql mc20.sql

70. SQL Model Clause 70Title of presentation

71. Part 5: Recursive Subquery Factoring

72. Recursive Subquery Factoring: Topics Subquery Factoring Concepts Recursive Examples Simulating Connect By Performance More recursive examples

73. Subquery Factoring Since version 9 Let’s you assign a name to a subquery block Modular Programming in SQL Also known as “WITH clause” or “Common Table Expressions” Second to last factored subquery: comma instead of “WITH” /*+ MATERIALIZE */ and /*+ INLINE */ Must use each factored subquery? rsf1.sql rsf2.sql rsf3.sql

74. Recursive Subquery Factoring: Concepts Since version 11.2 Let’s you query hierarchical data More powerful than CONNECT BY Anchor member UNION ALL recursive member Recursive member cannot contain: DISTINCT, Model clause, aggregate functions and analytic functions SEARCH DEPTH / BREADTH FIRST CYCLE rsf4.sql rsf5.sql rsf6.sql

75. Recursive Subquery Factoring: Examples Fibonacci fib(0) = 0 fib(1) = 1 fib(n+2) = fib(n+1) + fib(n) Interest and rates rsf7.sql rsf8.sql

76. Simulating Connect By LEVEL SYS_CONNECT_BY_PATH CONNECT_BY_ROOT CONNECT_BY_ISCYCLE CONNECT_BY_ISLEAF rsf9.sql rsf10.sql rsf11.sql rsf12.sql rsf13.sql

77. Recursive Subquery Factoring: Performance /*+ CONNECT_BY_FILTERING */ /*+ NO_CONNECT_BY_FILTERING */ rsf14.sql

78. More Recursive Examples Calculating die probabilities Exponential Moving Average X = (K * (C - P)) + P Where: X = Current EMA (i.e. EMA to be calculated) C = Current original data value K = Smoothing Constant P = Previous EMA Sudoku solver rsf15.sql rsf16.sql rsf17.sql

79. Recursive Subquery Factoring 79Title of presentation

80. Part 6: Regular Expressions

82. Regular Expressions: Topics Introduction Metacharacters POSIX Perl-based extensions More examples

83. Regular Expressions: Introduction A regular expression is a search pattern Since version 10: REGEXP_INSTR REGEXP_SUBSTR REGEXP_REPLACE REGEXP_LIKE Since version 11: REGEXP_COUNT re1.sql re2.sql re3.sql re4.sql re5.sql

84. Regular Expressions: Metacharacters Consists of metacharacters and character literals IEEE Portable Operating System Interface (POSIX) Oracle SQL Multilingual Extensions Oracle SQL Perl-influenced Extensions

85. Regular Expressions: POSIX SyntaxName.Any character +One or more *Zero or more ?Zero or one {m}Exact count {m,}At least count {m,n}Between count [char…]Matching character list [^char…]Non-matching char. list SyntaxName [a|b]Or (expr)Grouping \nBack Reference \Escape character ^Beginning-of-line anchor $End-of-line anchor [:class:]POSIX character class [.element.]Collating Element Operator [=char=] Character Equivalence Class re6.sql

86. Regular Expressions: Perl-based extensions re8.sql SyntaxDescriptionEquivalent to \dMatches a digit character[[:digit:]] \DMatches a non-digit character[^[:digit:]] \wMatches a word character (alphanumeric or _)[[:alnum:]_] \WMatches a non-word character[^[alnum:]_] \sMatches a whitespace character[[:space:]] \SMatches a non-whitespace character[^[:space:]] \AMatches beginning of string (multiline mode) \ZMatches end of string (multiline mode) \zMatches end of string (multiline mode)

87. Regular Expressions: Perl-based extensions re9.sql SyntaxDescription +?Matches one or more occurences of preceding expression non-greedy *?Matches zero or more occurences of preceding expression non-greedy ??Matches zero or one occurences of preceding expression non-greedy {m}?Matches exactly m occurences of preceding expression non-greedy {m,}?Matches at least m occurences of preceding expression non-greedy {m,n}?Matches between m and n occurences of preceding expr. non-greedy

88. Regular Expressions: Examples Formatting phone numbers Formatting names Retrieving information from large text More examples re12.sql re11.sql re10.sql re13.sql

89. Regular Expressions 89Title of presentation

90. Part 7: XML in SQL

91. XML in SQL: An opinion 19 136 0 8 56 Z02 E06 260

92. XML in SQL: Topics Composing XML from relational table data Shredding XML into relational format Some more possibilities

93. XML in SQL: Relational  XML The most important functions: XMLElement XMLForest XMLAgg XMLAttributes xis1.sql xis2.sql xis3.sql xis4.sql

94. XML in SQL: Relational  XML Other functions: XMLCast XMLCData XMLColAttVal XMLComment XMLConcat XMLDiff XMLExists XMLIsValid xis5.sql

95. XML in SQL: Relational  XML Other functions: XMLParse XMLPatch XMLPi XMLQuery XMLRoot XMLSerialize XMLTransform xis6.sql

96. XML in SQL: XML  Relational XMLSequence XMLTable xis7.sql xis8.sql

97. XML in SQL: More possibilities Dynamically evaluating expressions String aggregation XQuery expressions FLOWR (for, let, order by, where, return) xis9.sql xis10.sql xis11.sql

98. XML in SQL 98Title of presentation

99. Part 8: Frequently Occuring Problems

100. Frequently Occuring Problems Row / Number Generation Interval Based Row Generation Splitting Comma Separated Strings String Aggregation Pivoting Unpivoting Tabibitosan

101. Row / Number Generation fop1.sql

102. Interval Based Row Generation fop2.sql

103. Splitting Comma Separated Strings → fop3.sql

104. String Aggregation → fop4.sql

105. Pivoting fop5.sql

106. Unpivoting fop6.sql

107. Tabibitosan fop7.sql

108. Thank you for your attention 108 Title of presentation