Oracle University SQL Masterclass Rob van WijkJune 9 & 10, 2011, Tallinn, Estonia

About me Work with Oracle and SQL since 1995 From: Blog: Forums: Utrecht, Netherlands

Agenda Day 1 Part One:Do More With SQL and Joining Part Two:Analytic Functions Part Three:Grouping & Aggregating Part Four:SQL Model Clause Day 2 Part Five:Recursive Subquery Factoring Part Six:Regular Expressions Part Seven:XML in SQL Part Eight:Frequently Occuring SQL Problems

Part 1a: Do More With SQL

Goals As practical as possible As less regurgitating of documentation as possible Lots of example scripts Recognizable problems Do as much as possible in SQL and avoid shipping records for processing to PL/SQL or even Java at a middle tier.

If you want to build a ship, don't drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea. – Antoine de Saint Exupéry

One SQL engine versus …

… two engines. procedural engine SQL engine context swtiches dmws1.sql

You risk wrong results because of different start times of queries with default READ COMMITTED isolation level dmws2.sql

Part 1b: ANSI joins and partitioned outer join

ANSI joins: Comparison with Oracle-syntax CROSS JOIN INNER JOIN OUTER JOIN NATURAL JOIN aj3.sql aj2.sql aj1.sql aj4.sql

ANSI joins: Full Outer Join a FULL OUTER JOIN b ≡ a LEFT OUTER JOIN b UNION ALL b WHERE NOT EXISTS a 11g: Native full outer join _optimizer_native_full_outer_join /*+ NATIVE_FULL_OUTER_JOIN */ /*+ NO_NATIVE_FULL_OUTER_JOIN */ aj5.sql aj6.sql

ANSI joins: Partitioned Outer Join Outer join: NULL rows for missing values Partitioned outer join: NULL rows for missing values per, …, aj7.sql

ANSI joins 15Title of presentation

Part 2: Analytic Functions

Analytic Functions: Topics Introduction Mind set Evaluation order Main syntax Examples Window clause

Analytic Functions: Introduction Of every employee please show me: His name The department he’s working in His salary The cumulative salary per department Percentage of salary within the department Percentage of salary within the company where employees are sorted by department and salary af1c.sql af1b.sql af1a.sql

Analytic Functions: Introduction Since Enterprise Edition Look like well known aggregate functions like SUM, COUNT and AVG … but they don’t aggregate Prevents self joins Have been extended with new functions and new options in more recent versions af2.sql

Analytic Functions: Mind set Don’t think “rows” … EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO CLARK MANAGER KING PRESIDENT MILLER CLERK ADAMS CLERK FORD ANALYST JONES MANAGER SCOTT ANALYST SMITH CLERK ALLEN SALESMAN BLAKE MANAGER JAMES CLERK MARTIN SALESMAN TURNER SALESMAN WARD SALESMAN FORD ANALYST

Analytic Functions: Mind set … but think “sets” EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO CLARK MANAGER KING PRESIDENT MILLER CLERK ADAMS CLERK FORD ANALYST JONES MANAGER SCOTT ANALYST SMITH CLERK ALLEN SALESMAN BLAKE MANAGER JAMES CLERK MARTIN SALESMAN TURNER SALESMAN WARD SALESMAN EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO CLARK MANAGER KING PRESIDENT MILLER CLERK ADAMS CLERK FORD ANALYST JONES MANAGER SCOTT ANALYST SMITH CLERK ALLEN SALESMAN BLAKE MANAGER JAMES CLERK MARTIN SALESMAN TURNER SALESMAN WARD SALESMAN

Analytic Functions: Evaluation order Last Even after evaluating HAVING clause And after ROWNUM has been assigned But before ORDER BY clause Filtering on outcome of analytic function: nest the query using an inline view or use subquery factoring af3b.sql af3a.sql

Analytic Functions: Main syntax (,, …) OVER ( )

Analytic Functions: The functions LAGFIRST / LASTPERCENT_RANK LEADCOUNTPERCENTILE_DISC FIRST_VALUESUMPERCENTILE_CONT LAST_VALUEMAXCORR NTH_VALUEMINCOVAR_POP RANKAVGVARIANCE DENSE_RANKNTILEVAR_x (2 times) RATIO_TO_REPORTCUME_DISTSTDDEV_x (3 times) ROW_NUMBERLISTAGGREGR_x (9 times)

Analytic Functions: Partition clause PARTITION BY [, ]* to let the analytic function operate on a subset of the rows with the same values for the partition by expression values. af4.sql

Analytic Functions: Order By clause ORDER BY [ASC|DESC] [NULLS FIRST|NULLS LAST], … Its presence changes the default window of an analytic function from the total set to a running total. af5.sql

Analytic Functions: Example 1 Top N queries What do I mean exactly with: “Show me the top 3 earning employees per department” RANK DENSE_RANK ROW_NUMBER af6.sql

Analytic Functions: Example 2 1. David Zabriskie (USA)0.58:31 2. Ivan Basso (ITA) + 0:17 3. Paolo Savoldelli (ITA) + 0:44 4. Marzio Bruseghin (ITA) + 0:48 5. Serguei Gonchar (UKR) z.t. 6. Vladimir Karpets (RUS) + 1:07 7. Markus Fothen (GER) + 1:15 8. Thomas Dekker (NLD) + 1:23 9. Jan Hruska (CZE) + 1: Danilo di Luca (ITA) z.t. af7.sql

Analytic Functions: Example 3 Requirement: non-overlapping & consecutive periods Columns Startdate and maybe Enddate Optimize to retrieve current period Options: 1)No Enddate column and use correlated subquery 2)Enddate column and database trigger code to check requirement 3)No Enddate column and use analytic function af8.sql

Analytic Functions: Example 4 Bills can be of type “Prepayment” or “Settlement” Bill lines have an amount. Each customer pays a prepayment each month. The bill contains one bill line with the amount. Each customer receives once a year a settlement bill. How to calculate the previous prepayment amount? This is the amount before the last settlement bill. af9.sql

Analytic Functions: Example 5 TIME QUANTITY :22: :22: :22: :22: :22: :22: :23: :23: MIN(TIME) MAX(TIME) QUANTITY :22:01 12:22: :22:45 12:22: :23:12 12:23: af10.sql

Analytic Functions: Window clause Total set: ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING Anchored set / running aggregate: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ROW / RANGE af13.sql af11.sql af12.sql

Analytic Functions 33Title of presentation

Part 3: Grouping & Aggregating

aog1.sql

Grouping & Aggregating: Topics Introduction GROUPING SETS ROLLUP CUBE Combining and calculating Supporting functions Inner workings MIN/MAX … KEEP … (DENSE_RANK FIRST/LAST … )

Grouping & Aggregating: Grouping Sets (1) GROUP BY expr 1, …, expr n ≡ GROUP BY GROUPING SETS ( (expr 1, …, expr n ) ) aog2.sql

Grouping & Aggregating: Grouping Sets (2) GROUP BY GROUPING SETS ( (expr 11, …, expr 1n ), …, (expr x1, …, expr xm ) ) ≡ GROUP BY expr 11, … expr 1n UNION ALL … UNION ALL GROUP BY expr x1, …, expr xm aog3.sql

Grouping & Aggregating: ROLLUP (1) GROUP BY ROLLUP ( set 1, …, set n ) ≡ GROUP BY GROUPING SETS ( (set 1, …, set n ), (set 1, …, set n-1 ), …, set 1, () )

Grouping & Aggregating: ROLLUP (2) ROLLUP (set 1, …, set N ) with N ≥ 1 leads to N+1 GROUPING SETS

Grouping & Aggregating: ROLLUP (3) Example: GROUP BY ROLLUP ( (deptno), (job,mgr), (empno) ) ≡ GROUP BY GROUPING SETS ( (deptno,job,mgr,empno), (deptno,job,mgr), (deptno), () ) aog4.sql

Grouping & Aggregating: CUBE (1) GROUP BY CUBE ( set 1, …, set n ) ≡ GROUP BY GROUPING SETS (all possible combinations between () and (set 1, …, set n ) )

Grouping & Aggregating: CUBE (2) CUBE (set 1, …, set N ) with N ≥ 1 leads to 2 N GROUPING SETS

Grouping & Aggregating: CUBE (3) Follows Pascal’s triangle 0 sets X 1 set 2 sets 3 sets 4 sets

Grouping & Aggregating: CUBE (4) Example: GROUP BY CUBE ( (deptno), (job,mgr), (empno) ) ≡ GROUP BY GROUPING SETS ( (deptno,job,mgr,empno), (deptno,job,mgr), (deptno,empno), (job,mgr,empno), (deptno), (job,mgr), (empno), () ) aog5.sql

Grouping & Aggregating: Calculating (1) GROUP BY deptno, ROLLUP(empno) ?

Grouping & Aggregating: Calculating (2) GROUP BY deptno, ROLLUP(empno) ≡ GROUP BY GROUPING SETS (deptno), GROUPING SETS ( empno, () )

Grouping & Aggregating: Calculating (3) Cartesian product ! GROUP BY deptno, ROLLUP(empno) ≡ GROUP BY GROUPING SETS (deptno), GROUPING SETS ( (empno), () ) ≡ GROUP BY GROUPING SETS ( (deptno,empno), (deptno) ) aog6.sql

Grouping & Aggregating: Calculating (4) Question: How many grouping sets does the clause below yield? GROUP BY ROLLUP(deptno,job), CUBE(mgr,hiredate) aog7.sql

Grouping & Aggregating: Functions GROUPING GROUPING_ID GROUP_ID aog8.sql

Grouping & Aggregating: Inner working (1) SORT GROUP BY Versus HASH GROUP BY

Grouping & Aggregating: Inner working (2) NULL NULL NULL NULL incoming set grouping set ( (deptno,empno) ) grouping set ( () ) grouping set ( (deptno) ) SORT GROUP BY ROLLUP (DEPTNO,EMPNO) aog9.sql

Grouping & Aggregating: Inner working (3) SORT GROUP BY (deptno,job) GENERATE CUBE SORT GROUP BY (deptno,job) incoming set 14 rows 9 rows 36 rows 18 rows CUBE (DEPTNO,JOB) aog10.sql deptno null & job null deptno not null & job not nulldeptno not null & job null deptno null & job not null

Grouping & Aggregating: Inner working (4) LOAD AS SELECT (into input table) TABLE ACCESS FULL (EMP) TEMP TABLE TRANSFORMATION VIEW TABLE ACCESS FULL (output table) temporary input table SYS_TEMP_... temporary output table SYS_TEMP_... LOAD AS SELECT (into outputtable) HASH GROUP BY TABLE ACCESS FULL (input table) iterate as much times as there are grouping sets aog11.sql

Grouping & Aggregating: Inner working (5) Optimize towards a ROLLUP or CUBE execution, if possible? aog12.sql

Grouping & Aggregating: Agg. Functions (1) COUNT SUM AVG MAX MIN STDDEV VARIANCE LISTAGG aog13.sql

Grouping & Aggregating: Agg. Functions (2) MAX(…) KEEP (DENSE_RANK FIRST ORDER BY …) MAX(…) KEEP (DENSE_RANK LAST ORDER BY …) MIN(…) KEEP (DENSE_RANK FIRST ORDER BY …) MIN(…) KEEP (DENSE_RANK LAST ORDER BY …) aog14.sql

Grouping & Aggregating 58Title of presentation

Part 4: SQL Model Clause

SQL Model Clause: Topics Introduction Syntax Examples Performance Alternatives Conclusion

SQL Model Clause: Introduction Treat data as multidimensional arrays Complex calculations across rows Syntax which resembles logic programming (Prolog) Can prevent exporting data to external applications like Excel/Numbers No more several copies of data on several PC’s anymore

SQL Model Clause: Syntax (1) Divide columns in three groups: PARTITION, DIMENSION and MEASURES Every partition is a separate array Dimensions identify a cell in every partition Measures are the columns you want to (re-)calculate The rules tell you how the data is to be manipulated

SQL Model Clause: Examples A model clause that does nothing Adding an extra row to the result set RETURN UPDATED ROWS The difference between MEASURES and PARTITION mc4.sql mc3.sql mc2.sql mc1.sql

SQL Model Clause: Example from the doc mc5.sql

SQL Model Clause: More examples ANY CV() FOR Iterating Reference models Difference between NULL and NAV IS PRESENT, PRESENTV and PRESENTNNV mc11.sql mc10.sql mc9.sql mc8.sql mc7.sql mc6.sql mc12.sql

SQL Model Clause: Complete syntax MODEL [ ] [MAIN ] [PARTITION BY ( )] DIMENSION BY ( ) MEASURES ( ) [ ] [RULES] (,,.., ) ::= ::= RETURN {ALL|UPDATED} ROWS ::= [IGNORE NAV | [KEEP NAV] [UNIQUE DIMENSION | UNIQUE SINGLE REFERENCE] ::= [UPDATE | UPSERT | UPSERT ALL] [AUTOMATIC ORDER | SEQUENTIAL ORDER] [ITERATE ( ) [UNTIL ]] ::= REFERENCE ON ON ( ) DIMENSION BY ( ) MEASURES ( )

SQL Model Clause: Examples Financial spreadsheet Fibonacci OTN-question Interest and rates mc13.sql mc14.sql mc15.sql mc16.sql

SQL Model Clause: Performance Internal hash-tables in PGA Sequential Order  SQL MODEL ORDERED [FAST] Automatic Order  SQL MODEL [A]CYCLIC FAST  left side cell references are single cell references and aggregates at right side -if any- are simple arithmetic non-distinct aggregates, like SUM, COUNT, AVG and so on. mc17.sql

SQL Model Clause: Advanced examples Calculating die probabilities Exponential Moving Average X = (K * (C - P)) + P Where: X = Current EMA (i.e. EMA to be calculated) C = Current original data value K = Smoothing Constant P = Previous EMA Sudoku solver mc18.sql mc19.sql mc20.sql

SQL Model Clause 70Title of presentation

Part 5: Recursive Subquery Factoring

Recursive Subquery Factoring: Topics Subquery Factoring Concepts Recursive Examples Simulating Connect By Performance More recursive examples

Subquery Factoring Since version 9 Let’s you assign a name to a subquery block Modular Programming in SQL Also known as “WITH clause” or “Common Table Expressions” Second to last factored subquery: comma instead of “WITH” /*+ MATERIALIZE */ and /*+ INLINE */ Must use each factored subquery? rsf1.sql rsf2.sql rsf3.sql

Recursive Subquery Factoring: Concepts Since version 11.2 Let’s you query hierarchical data More powerful than CONNECT BY Anchor member UNION ALL recursive member Recursive member cannot contain: DISTINCT, Model clause, aggregate functions and analytic functions SEARCH DEPTH / BREADTH FIRST CYCLE rsf4.sql rsf5.sql rsf6.sql

Recursive Subquery Factoring: Examples Fibonacci fib(0) = 0 fib(1) = 1 fib(n+2) = fib(n+1) + fib(n) Interest and rates rsf7.sql rsf8.sql

Simulating Connect By LEVEL SYS_CONNECT_BY_PATH CONNECT_BY_ROOT CONNECT_BY_ISCYCLE CONNECT_BY_ISLEAF rsf9.sql rsf10.sql rsf11.sql rsf12.sql rsf13.sql

Recursive Subquery Factoring: Performance /*+ CONNECT_BY_FILTERING */ /*+ NO_CONNECT_BY_FILTERING */ rsf14.sql

More Recursive Examples Calculating die probabilities Exponential Moving Average X = (K * (C - P)) + P Where: X = Current EMA (i.e. EMA to be calculated) C = Current original data value K = Smoothing Constant P = Previous EMA Sudoku solver rsf15.sql rsf16.sql rsf17.sql

Recursive Subquery Factoring 79Title of presentation

Part 6: Regular Expressions

Regular Expressions: Topics Introduction Metacharacters POSIX Perl-based extensions More examples

Regular Expressions: Introduction A regular expression is a search pattern Since version 10: REGEXP_INSTR REGEXP_SUBSTR REGEXP_REPLACE REGEXP_LIKE Since version 11: REGEXP_COUNT re1.sql re2.sql re3.sql re4.sql re5.sql

Regular Expressions: Metacharacters Consists of metacharacters and character literals IEEE Portable Operating System Interface (POSIX) Oracle SQL Multilingual Extensions Oracle SQL Perl-influenced Extensions

Regular Expressions: POSIX SyntaxName.Any character +One or more *Zero or more ?Zero or one {m}Exact count {m,}At least count {m,n}Between count [char…]Matching character list [^char…]Non-matching char. list SyntaxName [a|b]Or (expr)Grouping \nBack Reference \Escape character ^Beginning-of-line anchor $End-of-line anchor [:class:]POSIX character class [.element.]Collating Element Operator [=char=] Character Equivalence Class re6.sql

Regular Expressions: Perl-based extensions re8.sql SyntaxDescriptionEquivalent to \dMatches a digit character[[:digit:]] \DMatches a non-digit character[^[:digit:]] \wMatches a word character (alphanumeric or _)[[:alnum:]_] \WMatches a non-word character[^[alnum:]_] \sMatches a whitespace character[[:space:]] \SMatches a non-whitespace character[^[:space:]] \AMatches beginning of string (multiline mode) \ZMatches end of string (multiline mode) \zMatches end of string (multiline mode)

Regular Expressions: Perl-based extensions re9.sql SyntaxDescription +?Matches one or more occurences of preceding expression non-greedy *?Matches zero or more occurences of preceding expression non-greedy ??Matches zero or one occurences of preceding expression non-greedy {m}?Matches exactly m occurences of preceding expression non-greedy {m,}?Matches at least m occurences of preceding expression non-greedy {m,n}?Matches between m and n occurences of preceding expr. non-greedy

Regular Expressions: Examples Formatting phone numbers Formatting names Retrieving information from large text More examples re12.sql re11.sql re10.sql re13.sql

Regular Expressions 89Title of presentation

Part 7: XML in SQL

XML in SQL: An opinion Z02 E06 260

XML in SQL: Topics Composing XML from relational table data Shredding XML into relational format Some more possibilities

XML in SQL: Relational  XML The most important functions: XMLElement XMLForest XMLAgg XMLAttributes xis1.sql xis2.sql xis3.sql xis4.sql

XML in SQL: Relational  XML Other functions: XMLCast XMLCData XMLColAttVal XMLComment XMLConcat XMLDiff XMLExists XMLIsValid xis5.sql

XML in SQL: Relational  XML Other functions: XMLParse XMLPatch XMLPi XMLQuery XMLRoot XMLSerialize XMLTransform xis6.sql

XML in SQL: XML  Relational XMLSequence XMLTable xis7.sql xis8.sql

XML in SQL: More possibilities Dynamically evaluating expressions String aggregation XQuery expressions FLOWR (for, let, order by, where, return) xis9.sql xis10.sql xis11.sql

XML in SQL 98Title of presentation

Part 8: Frequently Occuring Problems

Frequently Occuring Problems Row / Number Generation Interval Based Row Generation Splitting Comma Separated Strings String Aggregation Pivoting Unpivoting Tabibitosan

Row / Number Generation fop1.sql

Interval Based Row Generation fop2.sql

Splitting Comma Separated Strings → fop3.sql

String Aggregation → fop4.sql

Pivoting fop5.sql

Unpivoting fop6.sql

Tabibitosan fop7.sql

Thank you for your attention 108 Title of presentation