Jim Czuprynski OnX Enterprise Solutions March 7, 2016 Regression Analyzed, Tout de Suite: Leveraging Oracle 12c Database Real Application Testing (RAT) Suite Jim Czuprynski OnX Enterprise Solutions March 7, 2016

My Credentials 35+ years of database-centric IT experience Oracle ACE Director Oracle DBA since 2001 Oracle 9i, 10g, 11g OCP 100+ articles on databasejournal.com and ioug.org Regular speaker at Oracle OpenWorld, IOUG COLLABORATE, and OTN ACE Tours Oracle-centric blog (Generally, It Depends) Taught core Oracle DBA courses (G/I+RAC, Exadata, Performance Tuning, Data Guard) for Oracle University

Who We Are Solution Provider focused on Technology Solutions Comprehensive Service Offerings Extensive reach — U.S., Canada, and the U.K. $700M annual revenue with over $100M in Services Industry Certifications across a broad selection of best-in-class IT manufacturers and technologies Significant investment in technical staff and facilities to support our clients’ projects Solution Provider focused on Data Center Solutions Comprehensive Service Offerings Extensive reach — USA, Canada and Europe $700M annual revenue with over $100M in Services Industry Certifications across a broad selection of best-in-class IT manufacturers and technologies Significant investment in technical staff and facilities to support our clients’ projects

Our Agenda Real Application Testing (RAT): Concepts and Terminology SQL Performance Analyzer (SPA) Database Capture (DBC) and Database Replay (DBR) Oracle 12cR1 Enhancements Workload Intelligence Workload Scale-Up Consolidated Database Replay Leveraging Complementary Features and Use Cases Practical Performance Tuning Scenarios Database Migration Scenarios Q+A  

Real Application Testing: Concepts and Architecture

Regression Happens … Often at the Worst Time How can I determine if even making a minor change to my database environment will impact application performance? Different settings for initialization parameters Reversing a non-default setting can often cause performance deflection Different database versions or releases Remember, the Oracle query optimizer is “upgraded” as part of just about every new release New versions of hardware, storage, or operating system ASM vs. NFS Enterprise Systems vs. “home-grown” environments Linux vs. Solaris vs. HPUX vs. AIX

Regression Happens … Often at the Worst Time How can I stress-test my current or a future database environment beyond the point of no return? This is also known as testing to destruction Usually involves generating an overwhelming number of users / sessions / requests for similar data simultaneously May also require combining unanticipated simultaneous workloads that have yet to occur at the same time Identifying regression vectors and impacts may be extremely difficult Complex systems mean there’s often multiple interactions between components that are triggering regression Often degrades to finger-pointing between IT groups

RAT Suite: Poor Acronym. Great Toolset! SQL Performance Analyzer Analyzes SQL statements that comprise application workloads or subsets thereof Detects changes in statement execution plans because of: Different initialization parameters settings Different database or optimizer versions Different hardware, storage, or operating system Regressing SQL statements can be analyzed for better performance via SQL Tuning Advisor Aimed primarily at improving query performance, not DML

RAT Suite: Database Workload Capture Captures an entire application workload, including DML Workloads are captured in the precise order in which they occur so that they can be played back in that same order … or not Workloads can be filtered based on execution attributes Database Service, Module, Action, Program, User, Instance AWR data can be captured for later comparison with replays

Database Workload Capture These shadow workload capture processes record each user session’s activity against the production database with minimal overhead (usually < 2% extra CPU) Shadow Workload Capture Processes SCC WorkloadCapture Directory SCP Application Workloads DW + DSS OLTP BATCH + IDL PRODUCTION (11.2.0.4) +DATA DBMS_WORKLOAD_CAPTURE procedures will start, filter, and stop the capture of actual application workloads into shadow workload files containing binary representation of DML and queries

RAT Suite: Database Workload Replay Replays all or part of a captured application workload Statements are replayed against test system, including DML By default, workload is replayed in precisely the same order as it was recorded Workloads can be selectively ramped up: Overwhelming number of users / sessions / requests for same data Unexpected combination of simultaneous yet normal workloads This enables testing to destruction, i.e. beyond a system’s current capacity to execute the workload

Database Workload Replay TEST / QA (12.1.0.2) (rewound to before production capture point) +DATA WorkloadCapture Directory WRC Actually, there are two sets of captured workload files: the workloads themselves as well as the order in which transactions and queries originally occurred Workload Replay Clients (WRCs) read the captured workload files and play them back in original order (or not, if desired!) DBMS_WORKLOAD_REPLAY procedures will start, filter, and stop the replay of shadow workload files containing binary versions representing actual application workloads generated

Real Application Testing: New Features in 12cR1

Consolidated Replay Permits the combination of different workloads captured from different databases at different times into one consolidated workload. Use cases include: Same workload, but generated in different 10gR2, 11gR1, and 11gR2 environments Same workload generated on same database, but during completely different timeframes Different workloads generated on same database captured during wildly diverse conditions

Workload Scale-Up Enables the combination of workloads for simultaneous execution for potential “test to destruction” Schema Remapping allows replaying a single workload into multiple schemas simultaneously Time Shifting permits lining up different workloads so that their peak activity periods occur simultaneously Workload Folding separates a single workload captured over a longer duration into different sub-workloads, then playing those workloads back simultaneously

Workload Scale-Up: Examples Time Shifting: Combine 07:00 – 14:00 and 17:00 – 23:00 workloads into a single workload during replay Workload Folding: Cut 07:00 – 14:00 workload into three separate sub-workloads and then replay them simultaneously “Time-Shifted” Workload 17:00 – 23:00 07:00 – 14:00 “Folded” Workload

Workload Intelligence Workload Intelligence analyzes a captured workload for execution patterns that aren’t obvious to human brains Java-based program Requires creation of a separate schema for capturing results of analysis Returns one or more sets of SQL statements that comprise a workload pattern

Real Application Testing: Real-World Scenarios

Scenario #1: Upgrade from 11gR2 to 12cR1 Current Database Environment: Oracle 11gR2 (11.2.0.4) database on Microsoft Windows Supports primarily OLTP workload, some DSS, limited contention New Database Environment: Oracle 12cR1 (12.1.0.2) database on Oracle Linux A large non-partitioned table will now be partitioned Does the new environment handle current workloads? Queries against newly-partitioned table must perform 2X – 3X faster Do any existing queries demonstrate regression? Will placing tables in IMCS offer any benefits for analytic queries?

Scenario #2: Transitioning to Hybrid Cloud Current Database Environment: Oracle 11gR2 (11.2.0.4) database Supports purely data warehouse / analytic workloads Database server is “bare metal” Exadata X2-2 New Database Environment: Oracle 12cR1 (12.1.0.2) database, same application workloads Transitioning to Oracle Cloud DBaaS environment (Exadata X5-2) Does the new environment handle current workloads? Do any existing queries demonstrate regression? Is an Oracle Cloud Database Extreme Performance environment required to handle the application workload?

Scenario #3: Test to Destruction Current Database Environment: Oracle 11gR2 (11.2.0.4) database Oracle Enterprise Linux 5.6 virtualized via VirtualBox Supports both OLTP and DSS workloads with limited contention: Peak workload for OLTP between 10:00 - 11:00, 60 user sessions Peak workload for DSS between 19:00 – 21:00, 80 user sessions New Database Environment: Oracle 12cRx (12.2.x.x) database Oracle Enterprise Linux 7.2 virtualized via VMWare ESX Must support peak OLTP and DSS workloads simultaneously 50% more memory available Oracle In-Memory Option licensing warily considered CIO and CFO must know: Is this new environment sufficient?

Capturing a Workload 1 Create appropriate directory objects: 3 DROP DIRECTORY DBRCONTROL; CREATE DIRECTORY DBRCONTROL AS '/home/oracle/DBRControl'; GRANT READ, WRITE ON DIRECTORY DBRCONTROL TO PUBLIC; … allow the capture to continue until all desired workloads have been recorded … 3 BEGIN DBMS_WORKLOAD_CAPTURE.ADD_FILTER( fname => 'SVC_ADHOC' ,fattribute => 'SERVICE' ,fvalue => 'ADHOC'); . . . fname => 'SVC_TPCH' ,fvalue => 'TPCH'); DBMS_WORKLOAD_CAPTURE.START_CAPTURE( name => 'DBR_CAPTURE_100' ,dir => 'DBRCONTROL' ,duration => NULL ,default_action => 'EXCLUDE' ,auto_unrestrict => TRUE ,capture_sts => TRUE ,sts_cap_interval => 5); END; / Apply filters to exclude any undesired user sessions, then start the capture process … 2 … and then end the capture process once all workloads are complete: 4 BEGIN DBMS_WORKLOAD_CAPTURE.FINISH_CAPTURE( timeout => 90 ,reason => 'Termination (after 90 seconds)'); END; /

Consolidated Replay: Setup Create appropriate directory objects and new user account for replay clients: 1 Prepare the Replay Schedule from previously-captured workloads: 2 DROP DIRECTORY DBRCONS; CREATE DIRECTORY DBRCONS AS '/home/oracle/DBRControl'; GRANT READ, WRITE ON DIRECTORY DBRCONS TO PUBLIC; DROP DIRECTORY DBCOLTP; CREATE DIRECTORY DBCOLTP AS '/home/oracle/DBRControl/OLTP'; GRANT READ, WRITE ON DIRECTORY DBCOLTP TO PUBLIC; DROP DIRECTORY DBCDSS; CREATE DIRECTORY DBCDSS AS '/home/oracle/DBRControl/DSS'; GRANT READ, WRITE ON DIRECTORY DBCDSS TO PUBLIC; DROP USER wrc CASCADE; CREATE USER wrc IDENTIFIED BY wrc DEFAULT TABLESPACE users TEMPORARY TABLESPACE temp PROFILE DEFAULT QUOTA UNLIMITED ON users; GRANT CONNECT, DBA TO wrc; . . . cpt_id := DBMS_WORKLOAD_REPLAY.ADD_CAPTURE( capture_dir_name => 'DBCOLTP' ,start_delay_seconds => 0 ,stop_replay => FALSE ,take_begin_snapshot => TRUE ,take_end_snapshot => TRUE ,query_only => FALSE ); capture_dir_name => 'DBCDSS' DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE; END; / DECLARE cpt_id NUMBER; BEGIN DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE( capture_dir => 'DBCOLTP'); capture_dir => 'DBCDSS'); DBMS_WORKLOAD_REPLAY.SET_REPLAY_DIRECTORY( replay_dir => 'DBRCONS'); DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE ( replay_dir_obj => 'DBRCONS' ,schedule_name => 'TEST2DESTRUCT'); . . .

Consolidated Replay: Preparation and Startup Initialize the consolidated replay and remap connections as necessary: 3 BEGIN DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY( replay_name => 'DBR_CONS_100' ,schedule_name => 'TEST2DESTRUCT‘); DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION( connection_id => 1 ,replay_connection => '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ora12201)(PORT=1521)) (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=DSS)))'); . . . END; / BEGIN DBMS_WORKLOAD_REPLAY.PREPARE_CONSOLIDATED_REPLAY( synchronization => TRUE ,connect_time_scale => 0 ,think_time_scale => 0 ,think_time_auto_correct => FALSE ,capture_sts => TRUE ,sts_cap_interval => 5 ); END; / Prepare the Consolidated Replay: 4 Kick off the minimum necessary Workload Replay Clients: 5 $> wrc wrc/wrc mode=replay replaydir=/home/oracle/DBRControl $> wrc wrc/wrc mode=replay replaydir=/home/oracle/DBRControl BEGIN DBMS_WORKLOAD_REPLAY.SET_REPLAY_TIMEOUT(FALSE); DBMS_WORKLOAD_REPLAY.START_CONSOLIDATED_REPLAY; END; / Start the Consolidated Replay task: 6

Consolidated Replay: Monitoring via SQL*Plus SELECT id ,name ,schedule_name ,capture_id ,user_calls ,status ,divergence_load_status DLS ,TO_CHAR(start_time,'yyyy-mm-dd hh24:mi:ss') start_dtm ,TO_CHAR(end_time,'yyyy-mm-dd hh24:mi:ss') end_dtm ,duration_secs ,num_clients ,num_clients_done ,(dbtime/1000000) dbtime_ss ,(think_time/1000000) think_time_ss ,(network_time/1000000) network_time_ss ,pause_time FROM dba_workload_replays; SELECT id ,name ,dbname ,dbversion ,directory ,capture_id ,user_calls ,status ,TO_CHAR(prepare_time,'yyyy-mm-dd hh24:mi:ss') prepare_dtm ,synchronization ,connect_time_scale ,think_time_scale ,think_time_auto_correct TTAC ,scale_up_multiplier SCM ,default_action FROM dba_workload_replays; Think # of Replay Think Time Scale Rply Replay User Prepare Synch Connect Time Auto Up Default ID# Operation Calls Status Time ronized? TimeScale Scale Corre Multipler Action ---- ------------ ------- ------------ ----------- ----------- --------- ------ ----- --------- ----------- 167 DBR_CONS_100 1943044 COMPLETED 2016-02-29 SCN 0 0 FALSE 1 INCLUDE 18:22:04 157 DBR_CONS_100 1943044 COMPLETED 2016-02-29 SCN 0 0 FALSE 1 INCLUDE 16:17:57 113 DBR_CONS_100 1943044 COMPLETED 2016-02-28 SCN 0 0 FALSE 1 INCLUDE 17:36:42 102 DBR_CONS_100 1943044 COMPLETED 2016-02-28 SCN 0 0 FALSE 1 INCLUDE 17:00:50 # of Replay Replay Replay Total Clnt Think Netwk Pause Rply User Start End Time Clnt Sess DB Time Time Time Time ID# Calls Status Time Time (s) Sess Done (s) (s) (s) (s) ---- -------- --------------------- ----------- ------ ------ ------ ---------- ---------- ---------- --------- 167 1943044 COMPLETED 2016-02-29 2016-02-29 629 2 2 19667.10 162448.47 3486.92 0 18:23:04 18:33:33 157 1943044 COMPLETED 2016-02-29 2016-02-29 872 2 2 35494.17 201475.82 4222.84 0 16:18:33 16:33:05 113 1943044 COMPLETED 2016-02-28 2016-02-28 563 2 2 18411.01 174882.85 3701.86 0 17:37:12 17:46:35 102 1943044 COMPLETED 2016-02-28 2016-02-28 537 2 2 16963.72 168811.75 3852.33 0 17:01:22 17:10:19

Consolidated Replay: Monitoring via OEM 13c CC

Consolidated Replay: Results VARIABLE replay_report CLOB; VARIABLE capture_id NUMBER; VARIABLE replay_id NUMBER; BEGIN :replay_id := 167; :replay_report := DBMS_WORKLOAD_REPLAY.REPORT( replay_id => :replay_id ,format => 'TEXT'); END; / SET LONGCHUNKSIZE 1000 SET LONG 30000000 SET LINESIZE 2000 SET PAGESIZE 0 PRINT :replay_report . . . Replay Schedule Information Schedule TEST2DESTRUCT contains 2 captures: ---------------------------------------------------------- | Information | Capture 1 | | Name | DBR_CAPTURE_100 | | Status | COMPLETED | | Database Name | ORCL | | Database Version | 11.2.0.3.0 | | Start Time | 24-02-16 12:27:13 | | End Time | 24-02-16 12:32:48 | | Duration | 5 minutes 35 seconds | | DB Time | 7363.499 seconds | | User Calls | 356556 | | Average Active Sessions | 21.98 | | Directory Object | DBCOLTP | -------------------------------------------------------------------------- | DB Name | DB Id | Release | RAC | Replay Name | Replay Status | | NCDB122 | 1289746138 | 12.2.0.0.2 | NO | DBR_CONS_100 | COMPLETED | Replay Information -------------------------------------------------- | Information | Replay | | Name | DBR_CONS_100 | | Status | COMPLETED | . . . | AWR DB Id | 1289746138 | | AWR Begin Snap Id | 371 | | AWR End Snap Id | 376 | | PL/SQL Capture Mode | TOP_LEVEL | | Replay Schedule Name | TEST2DESTRUCT |

Consolidated Replay: AWR Delta Comparisons

Thank You

Thank You For Your Kind Attention! If you have any questions or comments, feel free to: E-mail me at Jim.Czuprynski@onx.com Follow my blog (Generally, It Depends): http://jimczuprynski.wordpress.com Follow me on Twitter (@JimTheWhyGuy) Connect with me on LinkedIn (Jim Czuprynski)