1. PayPal’s Custom AWR warehouse Samrat Roy
September 2014

2. Agenda Introduction Overview of Production Database Environment
Business and DBA requirement for AWR warehouse
Custom AWR warehouse as Current Solution
Current and Future Solution

3. About ebay inc and me
ebay inc is a global commerce platform and payments leader
Senior Engineer in ebay inc’s Database Engineering team
Working with Oracle Database for more than 10 years
Helped create and maintain the Custom AWR/DB metadata database

4. Production Environment
Mostly 11g RAC, ASM and Exadata
One of the busiest OLTP database
500+ database instances, from 10 – 130 TB each
5-10K processes per instance
Execs 100K/sec, Redo 5-10MB/sec
Growing workload 40% year-on-year
EM 12c monitors 100+ db targets including 5 critical databases running payment system, user data applications

5. Business and DBA Requirement for custom AWR Warehouse
Business Problem:
Find and fix database performance degradation proactively and post upgrade
Proactively add capacity (storage/compute) in a timely planned manner
Compare application schemas between stage and production
which when translated to DBA problem statements and solutions :
Monitor and capture W-o-W, (Week-over-Week) and M-o-M (Month-over-Month) deviation for DB metrics, enabling following reports
W-o-W deviation weekly and on-demand
SQL profiling for W-o-W deviation in execs, lio, pio, cpu, elapsed time
Daily ASH trend for significant wait events and W-o-W comparison
Daily W-o-W “Busy-ness” report from Time Model statistics for all databases
Track storage growth to generate storage TTL (Time to Live), compute add or storage uplift for Primary as well as Standbys
Follow GoldenGate trail from source to each downstream db
Schema comparison report between stage and production db
Compare AWR data pre/post upgrade for new database wholly or partially (moving only few schemas)
Also answer DBA type questions for ongoing operations, for e.g:
When table A was dropped, renamed or what column was added and when
Table “busyness” by time of day for scheduling table maintenance across both Production and Active DataGuard Standbys
Compare Initialization parameter, patch, compute, storage between Current Primary and Switchover Target
Global Database Metadata: Version, Type, Role, CPU, Memory, Storage (Primary/Standby, etc.)

6. Simple use cases SQL profile across all dbs for given sql_id @sqlbyid
Enter value for sql_id : 1nyxab
/* sample sql for sql_id 1nyxab*/SELECT * from A where col1=:col1;
SNAP_ID DBNAME EXECS_DELTA AVG_BG AVG_IO AVG_ROWS
DB_A_LIVE
DB_A_RO
DB_A_RO
Table profile across all dbs for given table
@sqlbytable
Enter value for table_name: TAB_A
DBNAME SQL_TYPE EXEC_PER_SNAPSHOT LIO_PER_SNAPSHOT PIO_PER_SNAPSHOT NUM_SQLS TOP_SQL_ID
DB_A_LIVE SELECT f8uagxysgjfh9
DB_A_RO SELECT nskqy7k2b26x
GG trail from SOR to all downstream copies for given table
./ggs_tab_path.pl <hostname> <schema> <tablename>
LEVEL 0 hostname ( _ OGGCORE_ _PLATFORMS_ ): (db) EXTRACT extr -> datapump:/oracle/gg/traildata/db/ja as of 26-SEP-14
LEVEL 1 datapump( _01 OGGCORE_ _PLATFORMS_ ): () EXTRACT extr1 -> downstream.db.paypal.com:/oracle/downstream/goldengate/db/ja as of 26-SEP-14
LEVEL dbhost1( _01 OGGCORE_ _PLATFORMS_ ): no extracts found to process the trail/table_owner/table_name combo
LEVEL dbhost2( _01 OGGCORE_ _PLATFORMS_ ): (downstreamdb2) REPLICAT replicat2 as of 26-SEP-14

7. Not-so-simple use case : Track health across all dbs W-o-W
Time Model Stats - WoW
========================================
DBNAME NAME PMDAY VALUE(Secs) % diff DELTA_DBTIME DB PCT%
DB_A_LIVE DB time TUESDAY % %
TUESDAY % %
TUESDAY % %
sql execute elapsed time TUESDAY % %
TUESDAY % %
TUESDAY % %
DB_B_LIVE DB time TUESDAY % %
TUESDAY % %
TUESDAY % %
sql execute elapsed time TUESDAY % %
TUESDAY % %
TUESDAY % %
column Contents
VALUE(Secs) The difference, in Seconds, of the given statistics trended for the given Time period.
% diff the difference , in percentage, of the given statistics trended for the given time period.
DELTA_DBTIME The difference between NOW and PREVIOUS DAY, of the "DB Time" statistic.
It would repeat for all the statistic for ease of comparison during manually reading the report
DB PCT % For "DB Time" statistic, this indicates the % difference between NOW and PREVIOUS DAY
for the other statistic, it is the VALUE of that statistic in % of DB Time (ie) All the statistics other than "DB Time" are expressed in percentage of "DB Time"
Other pointers for reading this report
==============================
This report shows jumps/reduction in DB Time for the trended period which can be used as a measure to check the TOPn report or the "Executions Stats Changes" report.
The profile of the "DB PCT"% for stats other than "DB Time" should remain fairly constant across the trended period. Any discernible change in that profile would mean that system is deviating from past behavior. If there are no discernible changes, then it would mean that the organic growth is in expected lines.

8. Drill down : Track top waits and SQL profile across all dbs
DBNAME EVENT_NAME time:HH WAITTIME WAITPCT
DB_A_LIVE enq: TX - row lock contention 09-SEP-2014 TUE %
16-SEP-2014 TUE %
23-SEP-2014 TUE %
Streams miscellaneous event 09-SEP-2014 TUE %
16-SEP-2014 TUE %
DBNAME EVENT_NAME time:HH WAITTIME WAITPCT
DB_B_LIVE db file sequential read SEP-2014 TUE %
16-SEP TUE %
23-SEP-2014 TUE %
SQL Profile Trend report - WoW
==============================
DBNAME DAY SQL_ID Ex% Buf% Elp% Cpu% Phy% Ex# Buf# Phy# Elp# Cpu#
DB_A_LIVE TUE dktb2hv6v9d
fbhuck3ryu
TUE 1kuyk21zwn1ac
dktb2hv6v9d
Significant wait events for the day
Only displays events which contributes > 2% of TOTAL Waits
WAITPCT pct contribution to TOTAL WAIT TIME for given day
WAITTIME secs waited/spent on that wait event

9. Further drill down on sql_id
Turns out SQL is a “SELECT-FOR-UPDATE” causing high enqueue waits impacting overall higher SQL execute elapsed time and higher CPU time
@sqlbyid
Enter value for sql_id : 1kuyk21zwn1ac
/* sample sql for sql_id 1kuyk21zwn1ac*/SELECT * from A FOR UPDATE;
SNAP_ID DBNAME EXECS_DELTA AVG_BG AVG_IO AVG_ROWS
DB_A_LIVE

10. Future OEM AWR Warehouse being considered
May use for performance tracking and integrated dashboard
Will need to extend the AWR Warehouse for functionality like ADG performance monitoring

11. Q & A
Thank You!