CERN IT Department CH-1211 Genève 23 Switzerland t SDC Stabilizing SQL execution plans in COOL using Oracle hints Andrea Valassi (IT-SDC) With many thanks to the Physics DBA team and especially Luca Canali for their expert help! CERN Oracle Tutorials, 5 th June 2013 Update of previous talks prepared with Romain Basset for the CERN DB Workshop in 2008 and the CERN ODF in 2010DB Workshop in 2008CERN ODF in 2010

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 2 Introduction COOL is a C++ software used by ATLAS and LHCb for storing and retrieving “detector conditions” data –Time variation and versioning of alignment, calibration and other data These data are read back by all reconstruction and analysis jobs The database schema and the SQL queries have been designed and maintained by the core COOL team –Users in ATLAS/LHCb instantiate several sets of tables with different table/column names following some centrally predefined patterns Oracle databases deployed at CERN and several Tier1’s –Differences in performance observed at different sites These “execution plan instabilities” have been solved in COOL using Oracle hints –Performance degradation initially seen in the 10g to 11g upgrade The COOL performance test suite was used to debug this regression –In this talk I will describe our experience with these and other issues

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 3 Outline COOL basics (only what is needed to understand the rest…) –Data model basics –Use case for this talk: “MV tags” (relational schema and SQL query) –Performance plots (how we define ‘good’ performance) Oracle performance optimization strategy –Basic SQL optimization (fix indexes and joins) –Execution plan instabilities (same SQL, different plans) Observe (causes: unreliable statistics, bind variable peeking) Analyze (10053 trace files and the BEGIN_OUTLINE block) Fix (rewrite queries to please the Optimizer; then add hints)

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 4 COOL – basics Conditions data –Detector data that vary in time and may be versioned –Several use cases (different schemas and SQL queries to optimize) Temperatures, voltages (measured – single version, SV) Calibration, alignment (computed – multiple versions, MV) COOL conditions objects (“IOV”s – interval of validity) –Metadata: channel (c), IOV (t since,t until ), version or tag (v) Predefined schema pattern for metadata for each use case –Data: user-defined “payload” (x1,x2,…) –Typical query: retrieve the condition data payload X that was valid at time T in channel C for tag V COOL relational implementation –Several backends (Oracle, MySQL, SQLite, FroNTier…) via CORAL C++ code with Oracle access via OCI (no PL/SQL procedures)

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 5 COOL – test case: MV tag retrieval Query: fetch all IOVs in [T1,T2] in tag PROD in all channels channelID Index2 since Index3 until Index4 tagID PK1 Index1 objectID PK2 pressuretemperatureobjectID PK 2. For each channel C, select IOVs in tag PROD in [T1, T2] (this is a very large table – and the most delicate part of the query to optimize) 3. For each IOV, fetch payload join channelNamechannelID PK join 1. Loop over channels

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 6 Testing query scalability When is SQL query performance “good” for us? –Query times should not grow as tables get larger Can we test this without creating huge tables each time? –We want a strategy for fast, cheap tests that can be repeated often For different use cases, for different versions of COOL and of Oracle Test slopes on small tables and extrapolate to large ones –“Good” performance is when there is no slope It should take the same time to fetch data from the top of a table (old ‘validity times” in IOVs) or from the bottom of a table (recent IOVs) In Oracle terms: no full scans of tables/indexes –Check the SQL execution plans for COOL queries

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 7 Is query time the same for all values of parameters T1, T2? –It was not in the initial COOL releases (≤ COOL 2.3.0) Query time is higher for more recent IOVs than for older IOVs "tagId=PROD AND chId=C AND ( (since ≤ T1< until) OR (T1 < since ≤ T2) )" Old COOL releases – poor SQL IOVs valid at t=T1 : inefficient use of index for query on two columns since and until (scan all IOVs with since ≤ T1, query time increases for high T1)

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 8 Basic optimization – better SQL In tag PROD, in each channel at most one IOV is valid at T1 –Build a better SQL strategy from this constraint (unknown to Oracle) The constraint is enforced in the C++ code, not in the database IOVs valid at t=T1 : efficient use of index (see reserve slides for details...) Problem fixed (?) (initial COOL231 candidate) Find s MAX = MAX(s) WHERE s<T1 in tag PROD and in the loop channel - Accept ( s = s MAX OR T1 < s ≤ T2) - Remove 'OR' using 'COALESCE'

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 9 Execution plan instabilities So, we thought the job was done... –Query time used to increase, we managed to make it flat But... every now and then our tests or our users reported performance issues again (...?...) –Example: different performance in ATLAS tests at CNAF and LYON Symptoms: same SQL, different execution plan –In time, we identified two possible causes for this: Bind variable peeking –Optimal exec plan for finding old IOVs and recent IOVs are different –Problem if optimal plan for old IOVs is used for finding recent IOVs Missing or unreliable statistics –Exec plan is not optimal because computed on wrong assumptions

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 10 Execution plan instabilities – plots Systematic study of 6 (2x3) cases –2 cases for bind variables: peek "low" (old IOVs) or "high" (recent IOVs) –3 cases for statistics: none, full, unreliable (computed on empty tables) Bad SQL (COOL230) Good SQL (COOL231), bad stats (empty tables). Good SQL (COOL231) and stats, peek 'low' (bad plan for 'high'). Good SQL (COOL231) and stats, peek 'high' (plan OK for all). Good SQL (COOL231), NO stats. Same 'good' SQL (COOL231), three different exec plans!

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 11 Analyze plans – trace files Look at the plan that was used for your query execution –More reliable than 'explain plan', 'set autotrace' and other methods... Look at how and why the Optimizer chose this plan –Bind variable values –Alternative plans attempted –Were user-supplied hints understood and used? The "Dumping Hints" section at the end Look at the Optimizer's outline for the most optimal plan –Get inspiration from the outline to prepare your user-supplied hints The "BEGIN_OUTLINE_DATA" section towards the end This section differs in the best plan and in sub-optimal plans

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 12 Stabilize plans – add hints This is an iterative process! In summary: –1. Execute your query for many cases (peek high/low...) –2. Get plan and outline for a case with good performance You want your plan to look like this in the end for all cases –3. Do you need some query rewrite? Are query blocks not named? Add QB_NAME and go to 1. Is Oracle rewriting your query? Change SQL and go to 1. Is Oracle using a different join order? Change SQL and go to 1. –4. Is there a case with bad performance? Get its outline. What is different in 'good' outline? Add as a hint and go to 1. Was your hint not used or not useful? Try another and go to 1. –5. Do all cases have good performance? You made it!

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL technicalities Generate a trace file 'myfile.trc' –From SQL*Plus ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1'; ALTER SESSION SET tracefile_identifier='myfile' –From CORAL: export CORAL_ORA_SQL_TRACE_ON="10053" export CORAL_ORA_SQL_TRACE_IDENTIFIER="myfile" Retrieve the trace file –Ask your friendly DBAs to get it from the server's udump... But please avoid generating (and asking for) trace files unless you need them... ;-) –To generate the COOL performance report we mount the files in read- only mode on a dedicated Oracle test server – not normally available

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL technicalities – exec plan flush You should invalidate existing exec plans between tests –To remove the effect of bind variable peeking (e.g. when testing the effect of different bind variable values) –To make sure that execution plans are recomputed and ORA trace files are as complete as possible To invalidate existing execution plans you may: –Flush the shared pool (DBA only – affects the whole DB) –Simpler hack: alter a relevant table in a dummy way e.g. “ALTER TABLE mytable LOGGING;”

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 15 Query rewrite – are you in control? Master your query blocks –Name your query blocks – syntax is “/*+ QB_NAME(xxx) */” Else the Optimizer will name them for you (e.g. “SEL$1”) –The Optimizer rewrites your query blocks? Do it yourself! Symptoms: query block names like “SEL$3F979EFD”, keywords like “MERGE” (remove inline views) or “CONCAT” (expand as union all) Solution: do what the Optimizer would do (e.g. remove MERGE by expanding subqueries in WHERE clause into normal joins) Master the order of your joins –The Optimizer reorders your joins? Do it yourself! Copy the Optimizer’s favorite order from the “LEADING” keyword

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 16 Stabilized plan – results Bad SQL (COOL230) Default hints added in COOL release –Stable good plans in all 6 cases (2 bind var peeking x 3 statistics) Good SQL (COOL231), good/missing/bad stats, peek 'low' or ‘high’ WITH HINTS Good SQL (COOL231) and stats, peek 'low' (bad plan for 'high') WITHOUT HINTS

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 17 Optimal query and hints Good plan with hints (peek low) Good plan with hints (peek low) Good plan with hints (peek low) Bad plan (peek low) Bad plan (peek low)

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 18 Hints vs. SQL Plan Baselines (1) We did evaluate SQL plan baselines in Oracle 11g –See Alex Loth’s talk at the CERN ODF 2010talk May be useful, but not the most appropriate for COOL –Too many different tables and queries in COOL to “load”/”accept” –Conversely, it is easy to add hints for all tables in COOL A single C++ method dynamically creates SQL for all use cases (SV, MV, user tags) After refactoring – previously each use case was optimized separately “Less to worry about” vs. “transparency and control”? –IMO: if you are the developer, you are in charge of SQL optimization You must master (i.e. get your hands dirty with) execution plans Relying on Oracle to do everything magically for you is not the best approach –SQL baselines and adaptive cursor sharing are both disabled in COOL We found that they could lead back to the loss of reproducibility we were fighting…

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 19 Plan BaselinesHints Control:in DBMS layerin application layer Maintenance:by DBAby developers SQL plan evolution:only improved plans are accepted better execution plans may remain unnoticed Manageability:via web user interface of Oracle Enterprise Manager modifications of source code are necessary must load baseline for each individual query may define hints once for similar queries Transparency:SQL execution becomes less transparent SQL execution is reproducible Fixing execution plans:may prevent better execution plans from showing up N/A A. Loth, CERN ODF 2010CERN ODF 2010 Hints vs. SQL Plan Baselines (2)

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 20 COOL automatic performance report The previous slides describe work done in More recently, more work on performance had to be done to debug issues in the 10g to 11g server upgrade The test procedure was streamlined, documented and fully automatized to more easily analyze performance –See A detailed report can be created with one command –Covering 9 common use cases for querying COOL data –Showing queries, hints, performance plots and execution plans –e.g.

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 21 COOL performance report ( )

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 22 Oracle 10g to 11g migration (1) No issue seen in early COOL tests on in Q But all seemed wrong when testing in Q4 2012! –Bad performance in ALL 6 curves without hints (e.g. with good stats too) –Bad performance in ALL 6 curves with hints too! –Need a complete rethink of SQL query strategy in COOL?

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL Luckily, the issue disappeared in Oracle ! –Problem seen in October 2011 is now understood to be caused by a bug in Oracle server (Oracle bug ) Confirmed by enabling/disabling Oracle patch in a private DB Bug was introduced in – it was not there in ! –Early COOL 11g tests in 2010 had seen no issue as they used –Bad news: even a minor server patch can break performance! Fixed in – good news: we should no longer worry now –Performance on is as good as on Execution plans look a bit different, but algorithm is probably the same COOL - 23 Oracle 10g to 11g migration (2)

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 24 COOL execution plans: 10g vs. 11g With hints: only one (good) plan on and –Plan looks different but is probably exactly the same algorithm Using INDEX RANGE SCAN (MIN/MAX) With hints: 3 (bad) plans on –Depending on statistics and bind variables (no exec plan stability!) All of them are bad and involve INDEX FULL SCAN (good) (good) (bad) – 1 st of 3 plans

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 25 Frontier and Squid caching The performance optimizations described so far were highly critical for many concurrent Oracle connections –This used to be the case a few years ago (e.g. CNAF/Lyon instability) Things are (slightly) more relaxed now thanks to the more widespread use of Frontier/Squid in ATLAS –Read-only data caching in Squid using http protocol –Connection multiplexing in hierarchy of Squid deployment –Used by CMS since several years, progressively adopted for many ATLAS use cases (e.g. distributed analysis, Tier0 production) HLT configuration also uses caching/multiplexing via the CORAL server –But Oracle SQL query optimization is still necessary because the Frontier server does connect to Oracle (and read-write users too)

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 26 CORAL components and their usage DB lookup XML COOL C++ API C++ code of LHC experiments (independent of DB choice) POOL C++ API use CORAL directly OracleAccess (CORAL Plugin) OCI C API CORAL C++ API (technology-independent) Oracle DB SQLiteAccess (CORAL Plugin) SQLite C API MySQLAccess (CORAL Plugin) MySQL C API MySQL DB SQLite DB (file) OCI (password, Kerberos) OCI FrontierAccess (CORAL Plugin) Frontier API CoralAccess (CORAL Plugin) coral protocol Frontier Server (web server) CORAL server JDBC http coral Squid (web cache) CORAL proxy (cache) coral http XMLLookupSvc XMLAuthSvc LFCReplicaSvc (CORAL Plugins) Authentication XML (file) LFC server (DB lookup and authentication) No longer used but minimally maintained Dropped in 2012 CORAL plugins interface to 5 back-ends -Oracle, SQLite, MySQL (commercial) -Frontier (maintained by FNAL) -CoralServer (maintained in CORAL) Maintained by ATLAS as of 2012 CORAL is used in ATLAS, CMS and LHCb in most of the client applications that access physics data stored in Oracle Oracle data are accessed directly on the master DBs at CERN (or their Streams replicas at T1 sites), or via Frontier/Squid or CoralServer ORACLE Hints in COOL - 26

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 27 Summary COOL strategy for optimizing Oracle performance –Basic SQL optimization (fix indexes and joins) –Execution plan instabilities (same SQL, different plans) Observe (causes: unreliable statistics, bind variable peeking) Analyze (10053 trace files and the BEGIN_OUTLINE block) Fix (rewrite queries to please the Optimizer; then add hints) Automatized creation of performance report Validation of query performance on Oracle database offload by caching in Frontier

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 28 Some useful references –NSS 2008 proceedings Optimization and test strategy, bind variable peeking, statistics –CHEP 2012 proceedings Automatic performance report, Oracle 11g upgrade –COOL twiki THANK YOU! QUESTIONS?

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 29 Reserve slides

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 30 COOL – relational schema (simplified) channelID Index2 since Index3 until Index4 tagID PK1 Index1 objectID PK2 pressuretemperatureobjectID PK IOV2TAG table IOV tableCHANNELS table FK channelNamechannelID PK Metadata (green) –System-controlled –Different sets of tables for different versioning modes (here: MV tags) Data payload (red) –User-defined schema –Different sets of tables for different data channel categories ('folders')

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 31 COOL – test case: MV tag retrieval Query: fetch all IOVs in [T1,T2] in tag PROD in all channels –1. Loop over all channels in table CHANNELS –2. For each channel, select IOVs from table IOV2TAG In tag PROD in [T1, T2] – this is the most complex part of the query Simplest (suboptimal): "(since ≤ T1< until) OR (T1 < since ≤ T2)" –3. For each selected IOV, fetch payload from table IOV channelID Index2 since Index3 until Index4 tagID PK1 Index1 objectID PK2 pressuretemperatureobjectID PK 2. For each channel, select IOVs from IOV2TAG 3. For each IOV, fetch payload join channelNamechannelID PK join 1. Loop over CHANNELS

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 32 COOL – measuring performance channelID Index2 since Index3 until Index4 tagID PK1 Index1 objectID PK2 IOVs valid from t>T1 to t ≤ T2 : efficient use of index IOVs valid at t=T1 : inefficient use of index for query on two columns since and until (scan all IOVs with since ≤ T1, query time increases for high T1) Simplest (suboptimal): "tagId=PROD AND chId=C AND ( (since ≤ T1< until) OR (T1 < since ≤ T2) )" Is query time the same for all values of parameters T1, T2? Not in releases ≤ COOL : query time increases for more recent IOVs

CERN Oracle Tutorials – 5 th June 2013A. Valassi – ORACLE Hints in COOL - 33 Basic optimization – better SQL In tag PROD, in each channel at most one IOV is valid at T1 –General definition of MV tags This constraint is enforced in the C++ code, not in the database –Find s MAX = MAX(s) WHERE s<T1 in tag PROD and the loop channel Accept ( s = s MAX OR T1 < s ≤ T2) Remove 'OR' using 'COALESCE' IOVs valid at t=T1 : efficient use of index Problem fixed (?) (initial COOL231 candidate)