© 2010 Quest Software, Inc. ALL RIGHTS RESERVED Integrating Workload Replay into Database Change Management Bert Scalzo Database Domain Expert / Oracle ACE

2 Author’s Books Coming Soon

3 Purpose Learn how one can rely on Benchmark Factory and actual captured database workload replay to more reliably ensure that normal database changes don’t degrade the user experience... Might save your reputation, sanity and job Note: Not Oracle specific, but examples here shown using BMF and Oracle trace file replay

4 Agenda Common challenges when dealing with database change Benefits of using database workload replay in change management methodology Performing database workload replay with Benchmark Factory The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Quest Software’s products remain at the sole discretion of Quest Software.

5 Database Change All applications & databases change over time Often changes are small & thus seemingly safe Sometimes changes are significant & very risky All change introduces risk – no matter the size! In fact studies have shown that the most “buggy” changes to code occur when less than 5 lines are modified (i.e. quick fixes are not very successful) Database changes work much the same: when you try “quick and simple” modifications without much thought, bad things generally happen ….

6 The Many Sources of Change Hardware upgrades, virtual resource re-allocation, or virtual machine relocation Software upgrades (e.g. new operating system or update to virtual machine software) Operating system and/or virtual machine setup, configuration and/or initialization parameters Database patches and/or upgrades (e.g > or Oracle 10gR2- >11gR2) Database setup, configuration and/or initialization parameters (e.g. SGA/PGA memory allocations) Database structural changes (e.g. add/drop/alter tables, columns, indexes and views) Storage nature changes (e.g. start using partitions, sub-partitions, index organized tables, etc) New database feature usage (e.g. start using 11G optional OEM “auto plan stability” features) Statistics and/or histogram collection frequency, granularity, parameters and/or method (SQL command vs. PL/SQL package call) Relative size of the database (e.g. most databases grow over time – some grow exponentially!)

7 Managing Change Measure of success: –Effective:Results AFTER = BEFORE –Efficient:Performance AFTER ~= BEFORE How to measure effectiveness –Unit testing –Integration testing How to measure efficiency –Stress Testing (too often a simple hardware stress scenario to prove that system can handle a “load”) –Workload replay: only way to assure real success …

8 What is Workload Replay Run the database application to generate transactions, and capture those transactions for replay on the same or a different system Focus the application transactions captured around that which has changed (i.e. a new business rule or process introduces new or modified code and database objects, therefore run & capture workload around that change) Also advisable to capture an overall workload scenario so that bad side effects can be found

9 Traditional DBA Approach Usually focused on just one or a few SQL statements of potential interest Too Much & Too Little

10 Too Much & Too Little Explained DBA most likely did not write the application code that’s now running slowly (i.e. efficiency) DBA may not fully understand the underlying business problem and all of its “requirements“ (i.e. effectiveness) #1+#2 = Tune with Uncertainty (“guestimate”) Multiple changed dimensions or aspects of the application and its database often rolled out all at once – overwhelming the lone DBA Never enough time – Needs fixed Yesterday!

11 Performance Pressure Points

12 Measure Cause & Effect >>

13 Options for Oracle Capture & Replay Oracle Real Application Testing (i.e. RAT) –Expensive (must purchase license for both source & target) –Generally captures the “entire“ database workload (however 11gR2 offers some new & useful replay filtering options) –Requires manually performing database backup & recovery Manual Trace File Capture via Toad DBA -> Replayed by a Benchmark Factory Project –Requires DBA to perform all the capture work manually – and this is not always easy (i.e. can be many, many steps) MOUSE (cute internal name for a new feature) –Automates the entire capture & replay process (so BMF will offer push button simplicity that even a caveman can do ) –Offers extensive filtering on both the capture & replay

14 RAT from Oracle New Features Training

15 RAT Base Pricing (minus discounts, etc) #CPU * #Cores * CPU Cost Factor * Processor License Cost + software Update/Support fee Must purchase for both source and target Example (upgrading a Xeon DB server): –Source:Dual E GHz 2 * 4 *.5 * $11,500 + $0.00 = $46,000 –Target:Dual E GHz 2 * 6 *.5 * $11,500 + $2,530= $71,530 –OEM Diagnostic Pack (Target Only) 2 * 6 *.5 * $5,000 +$1,100 = $61,100 –OEM Tuning Pack (Target Only) 2 * 6 *.5 * $5,000 +$1,100 = $61,100 –TOTAL Base Pricing = $239,730 !!! –Assume Typical 30% = $167,811 !!! This is in addition to Database’s base cost…

16 Basic BMF Architecture Six simple steps: 1. Manually setup to collect the trace files 2. Manually run the DB workload to capture 3. FTP trace files to Windows environment 4. Create BMF project based on trace files 5. BMF import (i.e. parse) the trace files 6. Run that project to replay the workload

17 Capture via Toad (Manual Effort) Need both of these so that BMD can replay trace files

18 Capture via DB Trigger (Automated) CREATE OR REPLACE TRIGGER AUTO_SET_TRACE AFTER LOGON ON DATABASE BEGIN IF (USER IN ('MOE','LARRY','CURLY')) /* ONLY TRACE FOR THE 3 STOOGES */ THEN EXECUTE IMMEDIATE 'ALTER SESSION SET tracefile_identifier = ''XXXXX'''; EXECUTE IMMEDIATE 'ALTER SESSION SET max_dump_file_size = ''10M'''; DBMS_SESSION.SESSION_TRACE_ENABLE (waits=>FALSE, binds=>TRUE); END IF; EXCEPTION WHEN OTHERS THEN RAISE_APPLICATION_ERROR(-10001,'AUTO_SET_TRACE ERROR'); END; You can write sophisticated filters to limit the number, size and naming (i.e. prefix) of all trace files collected during the capture

19 Inspect Trace Files via Toad Much like Winzip for Trace files

20 Create BMF Project for Replay

21 Set Some Replay Parameters Couple of Notes: - All the trace files need copied from the database server to the local BMF Windows PC(s) - The more and larger the trace files the longer it will take for BMF to parse and load them - BMF generally maintains transaction ordering and timings (but different hardware, software, and configurations may skew the results as expected)

22 Run the Replay Job So now we can replay or re- run the captured database workload at any time by starting this BMF project

23 I’m Poor & Lazy I don’t have $$$ for RAT (and OEM packs) I don’t want to manually do anything (ever) I want a very simple, push button, read my mind, totally automated solution that does not break the bank …. New BMF feature (Q4 – code name MOUSE) –Simple in concept & totally automates the process –Simple Capture Wizard (creates the BMF project) –Robust filtering options for both capture & replay –Merely replay the BMF project (it handles it all) –Zero percent manual effort (fully automatic!)

24 BMF / MOUSE Architecture Look Familiar ?

25 New BMF Option: Capture & Replay

26 Create a Capture

27 Capture Wizard Steps

28 Capture Wizard Steps

29 Capture Wizard Steps

30 Monitoring the Capture Manually Stop when Done or by Schedule

31 BMF Job Built for You – Just Run It

32 More information: Visit –Expert blogs, how-to-videos and more! Paper on topic – gy=8&prod=&prodfamily=&lochttp:// gy=8&prod=&prodfamily=&loc Benchmark Factory Community: – –Interact with BMF users and the product developers